An Approach to Assessing the Reliability of Information in Popular Science Discourse Based on the Analysis of Argumentation and Scientific Fact-Checking
Main Article Content
Abstract
This paper examines the challenges of assessing the reliability of information extracted from popular science texts. Popular science is known to be characterized by a free style of presentation, loose references to primary sources and expert opinions, and the presence of controversial statements and conclusions. Clarifying these components of the author's reasoning can significantly increase the credibility among critical readers or, conversely, refute the authors' assertions.
This paper proposes a new approach to assessing the reliability of information contained in popular science publications. This approach utilizes D. Walton's argumentation theory to identify potentially unreliable claims and logical fallacies. Searching scientific databases and calculating trust metrics for scientific publications based on their metadata and scientific ratings will enable the verification of controversial claims and an assessment of their reliability. The use of large language models will enable the generation of explanations for decisions made.
To support the development and validation of this approach, two data collection methods are proposed. The first uses existing corpora of Russian-language popular science texts and commentaries with annotated argumentation. The annotation includes, among other things, polemical argumentation schemes that challenge the author's thesis or argument. The second method utilizes LLM for generating synthetic theses and popular science articles, based on actual scientific publications.
Article Details
References
2. Guo Z., Schlichtkrull M., Vlachos A. A Survey on Automated Fact-Checking // Transactions of the Association for Computational Linguistics. 2022. Vol. 10. P. 178–206. https://doi.org/10.1162/tacl_a_00454
3. Vykopal I., Pikuliak M., Ostermann S., Šimko M. Generative Large Language Models in Automated Fact-Checking: A Survey // arXiv preprint arXiv:2407.02351. arXiv, 2026.
4. Kumar S. et al. SciClaimHunt: A Large Dataset for Evidence-based Scientific Claim Verification // 2025 International Joint Conference on Neural Networks (IJCNN). Rome, Italy: IEEE, 2025. P. 1–10. https://doi.org/10.1109/IJCNN64981.2025.11227296
5. Vatolin A. et al. ruSciFact: Open Benchmark for Verifying Scientific Facts in Russian // Computational Linguistics and Intellectual Technologies. 2025. P. 2075–7182. https://doi.org/10.28995/2075-7182-2025-23-435-459
6. Min S. et al. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation // Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing / ed. Bouamor H., Pino J., Bali K. Singapore: Association for Computational Linguistics, 2023. P. 12076–12100. https://doi.org/10.18653/v1/2023.emnlp-main.741
7. Sitaula N. et al. Credibility-Based Fake News Detection // Disinformation, Misinformation, and Fake News in Social Media: Emerging Research Challenges and Opportunities / ed. Shu K., Wang S., Lee D., Liu H. Cham: Springer International Publishing, 2020. P. 163–182. https://doi.org/10.1007/978-3-030-42699-6_9
8. Chrysidis Z., Papadopoulos S.-I., Papadopoulos S., Petrantonakis P.C. Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checking // 3rd ACM International Workshop on Multimedia AI against Disinformation. 2024. P. 73–81. https://doi.org/10.1145/3643491.3660278
9. Visser J., Lawrence J., Reed C. Reason-checking fake news // Commun. ACM. 2020. Vol. 63, No. 11. P. 38–40. https://doi.org/10.1145/3397189
10. Glockner M., Hou Y., Nakov P., Gurevych I. Missci: Reconstructing Fallacies in Misrepresented Science // Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) / ed. Ku L.-W., Martins A., Srikumar V. Bangkok, Thailand: Association for Computational Linguistics, 2024. P. 4372–4405. https://doi.org/10.18653/v1/2024.acl-long.240
11. Schlichtkrull M., Guo Z., Vlachos A. AVERITEC: a dataset for real-world claim verification with evidence from the web // Proceedings of the 37th International Conference on Neural Information Processing Systems. Red Hook, NY, USA: Curran Associates Inc., 2023. P. 65128–65167. https://doi.org/10.52202/075280-2842
12. Wang W.Y. “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection // Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) / ed. Barzilay R., Kan M.-Y. Vancouver, Canada: Association for Computational Linguistics, 2017. P. 422–426. https://doi.org/10.18653/v1/P17-2067
13. Saakyan A., Chakrabarty T., Muresan S. COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic // Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) / ed. Zong C., Xia F., Li W., Navigli R. Online: Association for Computational Linguistics, 2021. P. 2116–2129. https://doi.org/10.18653/v1/2021.acl-long.165
14. Liu H. et al. Retrieval augmented scientific claim verification // JAMIA Open. 2024. Vol. 7, № 1. P. ooae021. https://doi.org/10.1093/jamiaopen/ooae021
15. Kotonya N., Toni F. Explainable Automated Fact-Checking for Public Health Claims // Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics, 2020. P. 7740–7754. https://doi.org/10.18653/v1/2020.emnlp-main.623
16. Wadden D. et al. Fact or Fiction: Verifying Scientific Claims // Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) / ed. Webber B., Cohn T., He Y., Liu Y. Online: Association for Computational Linguistics, 2020. P. 7534–7550. https://doi.org/10.18653/v1/2020.emnlp-main.609
17. Glockner M., Hou Y., Nakov P., Gurevych I. Missci: Reconstructing Fallacies in Misrepresented Science // Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) / ed. Ku L.-W., Martins A., Srikumar V. Bangkok, Thailand: Association for Computational Linguistics, 2024. P. 4372–4405. https://doi.org/10.18653/v1/2024.acl-long.240
18. Glockner M., Hou Y., Nakov P., Gurevych I. Grounding Fallacies Misrepresenting Scientific Publications in Evidence // Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) / ed. Chiruzzo L., Ritter A., Wang L. Albuquerque, New Mexico: Association for Computational Linguistics, 2025. P. 9732–9767. https://doi.org/10.18653/v1/2025.naacl-long.491
19. Walton D. Argumentation theory: A very short introduction //Argumentation in artificial intelligence. Boston, MA: Springer US, 2009. P. 1–22. https://doi.org/10.1007/978-0-387-98197-0_1
20. Akhmadeeva I., Kononenko I., Sidorova E., Shestakov V. Using Rhetorical Structures to Analyze Argumentation in Scientific Communication Texts // Computational Linguistics and Intellectual Technologies. 2025. P. 1–11. https://doi.org/ 10.28995/2075-7182-2025-23-1-11
21. Chistova E. End-to-End Argument Mining over Varying Rhetorical Structures // Findings of the Association for Computational Linguistics: ACL 2023 / ed. Rogers A., Boyd-Graber J., Okazaki N. Toronto, Canada: Association for Computational Linguistics, 2023. P. 3376–3391. https://doi.org/10.18653/v1/2023.findings-acl.209
22. Sidorova E.A. et al. Enriching the ontology of argumentation based on critical questions // 2025 IEEE XVII international scientific and technical conference on actual problems of electronic instrument engineering (APEIE). 2025. P. 1–5. https://doi.org/10.1109/APEIE66761.2025.11289342
23. Sery A. S. Data credibility when populating ontologies and knowledge graphs // Ontology of Designing. 2023. Vol. 13, № 1. P. 113–124. https://doi.org/ 10.18287/2223-9537-2023-13-1-113-124
24. Ilina D.V., Timofeeva M.K., Kononenko I.S., Sidorova E.A. Generating Critical Questions for D. Walton’s Argumentation Schemes// NSU Vestnik. Series: Linguistics and Intercultural Communication. 2026. Vol. 4, No. 1. P. 87–101. (In Russ.) https://doi.org/10.25205/1818-7935-2026-24-1-87-101
This work is licensed under a Creative Commons Attribution 4.0 International License.
Presenting an article for publication in the Russian Digital Libraries Journal (RDLJ), the authors automatically give consent to grant a limited license to use the materials of the Kazan (Volga) Federal University (KFU) (of course, only if the article is accepted for publication). This means that KFU has the right to publish an article in the next issue of the journal (on the website or in printed form), as well as to reprint this article in the archives of RDLJ CDs or to include in a particular information system or database, produced by KFU.
All copyrighted materials are placed in RDLJ with the consent of the authors. In the event that any of the authors have objected to its publication of materials on this site, the material can be removed, subject to notification to the Editor in writing.
Documents published in RDLJ are protected by copyright and all rights are reserved by the authors. Authors independently monitor compliance with their rights to reproduce or translate their papers published in the journal. If the material is published in RDLJ, reprinted with permission by another publisher or translated into another language, a reference to the original publication.
By submitting an article for publication in RDLJ, authors should take into account that the publication on the Internet, on the one hand, provide unique opportunities for access to their content, but on the other hand, are a new form of information exchange in the global information society where authors and publishers is not always provided with protection against unauthorized copying or other use of materials protected by copyright.
RDLJ is copyrighted. When using materials from the log must indicate the URL: index.phtml page = elbib / rus / journal?. Any change, addition or editing of the author's text are not allowed. Copying individual fragments of articles from the journal is allowed for distribute, remix, adapt, and build upon article, even commercially, as long as they credit that article for the original creation.
Request for the right to reproduce or use any of the materials published in RDLJ should be addressed to the Editor-in-Chief A.M. Elizarov at the following address: amelizarov@gmail.com.
The publishers of RDLJ is not responsible for the view, set out in the published opinion articles.
We suggest the authors of articles downloaded from this page, sign it and send it to the journal publisher's address by e-mail scan copyright agreements on the transfer of non-exclusive rights to use the work.