Enviar artigo por via de e-mail Automatic information extraction from ego-documents: a comparative analysis of the effectiveness of large language models based on the example of K.A. Berezkin's diary.
- Autores: Kuznetsov A.V.1
-
Afiliações:
- Edição: Nº 3 (2025)
- Páginas: 99-127
- Seção: Articles
- URL: https://bakhtiniada.ru/2585-7797/article/view/361946
- DOI: https://doi.org/10.7256/2585-7797.2025.3.75850
- EDN: https://elibrary.ru/ZAYBBF
- ID: 361946
Citar
Texto integral
Resumo
The subject of the study is a comparative analysis of the performance, analytical strategies, and limitations of four large language models – Gemini-2.5-Pro, o3, Grok3, and Deepseek-v3 – in the task of extracting structured information from a historical ego-document. The analysis aims to determine the models' ability to work with complex narratives characterized by a high degree of subjectivity, an abundance of indirect evidence, multi-layered meanings, and emotional coloration. The key limitations of the models – over-interpretation, missing indirect evidence, and the trade-off between completeness and accuracy – are considered part of their analytical strategies. The material used was the diary of the Vologda gymnasium student K.A. Berezkin for the year 1849. The work addresses a complex task of developing and testing an approach that allows for the transformation of unstructured source text into a dataset suitable for solving a specific historiographical task – analyzing the perception of the European revolutions of 1848-1849 in the Russian province. The methodology is based on the automatic extraction of structured information using large language models. A comprehensive toolkit has been developed, including a domain-specific ontology, prompts, and a detailed JSON schema for data capture. The performance of the models was evaluated based on quantitative (completeness, accuracy, F1-score) and qualitative indicators (granularity, adherence to the ontology, understanding of historical context, typical errors). The scientific novelty lies in the first systematic testing and comparative analysis of the performance of leading language models in working with a historical ego-document in domestic historiography. It was established that the models implement various data extraction strategies: from exhaustive, but "noisy" coverage (Gemini-2.5-Pro) to highly accurate, but selective (Deepseek-v3), which directly determines the suitability of the resulting dataset for different research scenarios: from exploratory analysis to the creation of verified databases. The key conclusion of the study is that automated extraction is not merely a technical operation, but a form of digital hermeneutics. Accordingly, the final dataset is not objective data passively "discovered" in the source, but capta – a set of information selected for a specific task. The study shows that the application of artificial intelligence raises historian's requirements for critical expertise, shifting their role from information retrieval to verification and interpretation of machine results.
Bibliografia
Авербух Р. А. Царская интервенция в борьбе с венгерской революцией 1848–1849. – Москва : Соцэкгиз, 1935. – 232 с. Березкин К. А. Дневник [Электронный ресурс] / подгот. текста: А. Завина, А. Микерин // Электронный корпус "Прожито". – URL: https://corpus.prozhito.org/person/925 (дата обращения: 08.09.2025). Володин А. Ю. Исторические исследования в контексте датаизма: методологический аспект // Вестник Пермского университета. История. – 2023. – № 4(63). – С. 135-147. – doi: 10.17072/2219-3111-2023-4-135-147. – EDN: GFSIOJ. Володин А. Ю. Цифровая герменевтика исторического источника: формализация как толкование // Вестник Пермского университета. История. – 2025. – № 2(69). – С. 87-100. – doi: 10.17072/2219-3111-2025-2-87-100. – EDN: PWRHKI. Гросул В. Я. Русские участники зарубежных революций первой половины XIX века // Новая и новейшая история. – 2007. – № 4. – С. 21-40. – EDN: IAMEKF. Кузнецов А. В. Большие языковые модели как инструмент историка // Новые информационные технологии в образовании и науке. – 2024. – № 13. – С. 75-92. – doi: 10.17853/2587-6910-2024-13-75-92. – EDN: QDSUHV. Кузнецов А.В. За пределами тематического моделирования: анализ исторического текста с помощью больших языковых моделей // Историческая информатика. 2024. № 4. С. 47-65. doi: 10.7256/2585-7797.2024.4.72560 EDN: UOIKPJ URL: https://nbpublish.com/library_read_article.php?id=72560 Линькова Е. В. Внешнеполитический курс Николая І в оценках отечественных консерваторов середины XIX в. // Вестник РУДН. Серия: История России. – 2014. – № 1. – С. 17-23. – EDN: RZVVTB. Медведева О. Э. Николай I и европейские революции в историографии // Вестник научной ассоциации студентов и аспирантов исторического факультета Пермского государственного гуманитарно-педагогического университета. Серия: Studis historica juvenum. – 2017. – № 1 (13). – С. 132-136. Международные отношения на Балканах, 1830–1856 гг. / [Г. Л. Арш, И. С. Варта, В. Н. Виноградов и др.] ; отв. ред. В. Н. Виноградов ; АН СССР, Ин-т славяноведения и балканистики. – Москва : Наука, 1990. – 349 с. Мещерякова А. О. Русские консерваторы и европейские революции 1848–1849 годов // Тетради по консерватизму. – 2017. – № 2. – С. 61-67. – EDN: YMFTLR. Орлик И. И. Венгерская революция 1848–1849 годов и Россия // Новая и новейшая история. – 2008. – № 2. – С. 21-40. – EDN: IRGMBF. Орлик О. В. Передовая Россия и революционная Франция (первая половина XIX века). – Москва : Наука, 1973. – 299 с. Стыкалин А. С. Правда и вымысел о реакции российского общества на венгерскую революцию 1848 г. Венгерская кампания 1849 г. и капитан Гусев // Историческая экспертиза. – 2014. – № 1. – С. 38-54. – EDN: WCHJSL. Федоров А. В. Отношение передовых людей России к Венгерской революции 1848–1849 годов // Вопросы истории. – 1957. – № 2. – С. 89-96. Alhamed F., Ive J., Specia L. Using large language models (LLMs) to extract evidence from pre-annotated social media data // Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024). – 2024. – P. 232-237. Bernard G. Détection et suivi d'événements dans des documents de presse historiques : дис. – Université de La Rochelle, 2022. De Toni F. et al. Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0 // BigScience 2022-International Workshop on Challenges & Perspectives in Creating Large Language Models 2022. – 2022. – P. 75-83. Drucker J. Humanities Approaches to Graphical Display [Электронный ресурс] // Digital Humanities Quarterly. – 2011. – Vol. 5, № 1. – URL: http://www.digitalhumanities.org/dhq/vol/5/1/000091/000091.html (дата обращения: 08.09.2025). Gao T., Langlais P. RaTE: a Reproducible automatic Taxonomy Evaluation by Filling the Gap // Proceedings of the 15th International Conference on Computational Semantics (IWCS 2023). – 2023. – P. 173-182. Goloviznina V. et al. Do LLMs Understand Why We Write Diaries? A Method for Purpose Extraction and Clustering // arXiv preprint arXiv:2506.00985. – 2025. – URL: https://arxiv.org/abs/2506.00985 (дата обращения: 08.09.2025). González-Gallardo C. E. et al. Leveraging open large language models for historical named entity recognition // International Conference on Theory and Practice of Digital Libraries. – Cham : Springer Nature Switzerland, 2024. – P. 379-395. Han R. et al. An empirical study on information extraction using large language models // arXiv preprint. – 2023. – arXiv:2305.14450. – URL: https://arxiv.org/abs/2305.14450 (дата обращения: 08.09.2025). Hauser J. et al. Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM) // Advances in Neural Information Processing Systems. – 2024. – Т. 37. – P. 32336-32369. Hiltmann T. et al. NER4all or Context is All You Need: Using LLMs for low-effort, high-performance NER on historical texts. A humanities informed approach // arXiv preprint. – 2025. – URL: https://arxiv.org/abs/2502.04351 (дата обращения: 08.09.2025). Hiltmann T. Hermeneutik in Zeiten der KI: Large Language Models als hermeneutische Instrumente in den Geschichtswissenschaften // KI:Text: Diskurse über KI-Textgeneratoren / ed. by G. Schreiber, L. Ohly. – Berlin ; Boston : De Gruyter, 2024. – S. 201-232. König M. ChatGPT und Co. in den Geschichtswissenschaften – Grundlagen, Prompts und Praxisbeispiele [Электронный ресурс] // Digital Humanities am DHIP. – 2024. – 19 Aug. – URL: https://dhdhi.hypotheses.org/9197 (дата обращения: 08.09.2025). Laato J. et al. Extracting Social Connections from Finnish Karelian Refugee Interviews Using LLMs // arXiv preprint. – 2025. – arXiv:2502.13566. – URL: https://arxiv.org/abs/2502.13566 (дата обращения: 08.09.2025). Lai V. D. et al. Event extraction from historical texts: A new dataset for black rebellions // Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. – 2021. – P. 2390-2400. Lavin M. Why digital humanists should emphasize situated data over capta [Электронный ресурс] // Digital Humanities Quarterly. – 2021. – Vol. 15, № 2. – URL: http://www.digitalhumanities.org/dhq/vol/15/2/000556/000556.html (дата обращения: 08.09.2025). Li J. et al. DiaryHelper: Exploring the use of an automatic contextual information recording agent for elicitation diary study // Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. – 2024. – P. 1-16. Pan H. et al. Taxonomy-Driven Knowledge Graph Construction for Domain-Specific Scientific Applications // Findings of the Association for Computational Linguistics: ACL 2025. – 2025. – С. 4295-4320. Piskorski J., Yangarber R. Information extraction: Past, present and future // Multi-source, multilingual information extraction and summarization. – Berlin, Heidelberg : Springer Berlin Heidelberg, 2012. – P. 23-49. Santini C. Combining language models for knowledge extraction from Italian TEI editions // Frontiers in Computer Science. – 2024. – Т. 6. – P. 1472512. Shin D. et al. Using large language models to detect depression from user-generated diary text data as a novel approach in digital mental health screening: instrument validation study // Journal of Medical Internet Research. – 2024. – Т. 26. – P. e54617. – doi: 10.2196/54617. – EDN: VFSRAP. Thaller M. Historical Information Science: Is There such a Thing? New Comments on an old Idea // Seminario Discipline Umanistiche e Informatica. Il Problema dell'Integrazione/Ed. T. Orlandi. Roma. – 1993. – P. 51-86. Tang X. et al. CHisIEC: An Information Extraction Corpus for Ancient Chinese History // Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). – 2024. – P. 3192-3202. Xie Y., La Mela M., Tell F. Multimodal LLM-assisted Information Extraction from Historical Documents: The Case of Swedish Patent Cards (1945–1975) and ChatGPT // The 9th Digital Humanities in the Nordic and Baltic Countries Conference (DHNB 2025), March 5-7, 2025, Tartu, Estonia. – University of Oslo Library, 2025. – P. 1-15. Xu D. et al. Large language models for generative information extraction: A survey // Frontiers of Computer Science. – 2024. – Т. 18. – № 6. – P. 186357. Zhang Y. et al. Teleclass: Taxonomy enrichment and llm-enhanced hierarchical text classification with minimal supervision // Proceedings of the ACM on Web Conference 2025. – 2025. – P. 2032-2042.
Arquivos suplementares
