Procedure for Comparing Text Recognition Software Solutions For Scientific Publications by the Quality of Metadata Extraction
Main Article Content
Abstract
Metadata of scientific publications are used to build catalogs, determine the citation of publications, and perform other tasks. Automation of metadata extraction from PDF files provides means to speed up the execution of the designated tasks, while the possibility of further use of the obtained data depends on the quality of extraction. Existing software solutions were analyzed, after which three of them were selected: GROBID, CERMINE, ScientificPdfParser. A procedure for comparing software solutions for recognizing texts of scientific publications by the quality of metadata extraction is proposed. Based on the procedure, an experiment was conducted to extract 4 types of metadata (title, abstract, publication date, author names). To compare software solutions, a dataset of 112,457 publications divided into 23 subject areas formed on the basis of Semantic Scholar data was used. An example of choosing an effective software solution for metadata extraction under the conditions of specified priorities for subject areas and types of metadata using a weighted sum is given. It was determined that for the given example CERMINE shows efficiency 10.5% higher than GROBID and 9.6% higher than ScientificPdfParser.
Article Details
References
2. Liu X., Zhang J., Guo C. Full‐text citation analysis: A new method to enhance scholarly networks //Journal of the American Society for Information Science and Technology. 2013. Т. 64. №. 9. P. 1852-1863.
3. Saier T., Färber M. unarXive: a large scholarly data set with publications’ full-text, annotated in-text citations, and links to metadata // Scientometrics. 2020. Vol. 125. No. 3. P. 3085-3108.
4. Safder I. et al. Deep learning-based extraction of algorithmic metadata in full-text scholarly documents // Information processing & management. 2020. Vol. 57. No. 6. P. 102269.
5. O’Leary N. A. et al. Exploring and retrieving sequence and metadata for species across the tree of life with NCBI Datasets // Scientific data. 2024. Vol. 11. No. 1. P. 732.
6. Safder I., Hassan S. U. Bibliometric-enhanced information retrieval: a novel deep feature engineering approach for algorithm searching from full-text publications // Scientometrics. 2019. Vol. 119. P. 257-277.
7. Joshi B., Symeonidou A., Danish S.M., Hermsen F. An End-to-End Pipeline for Bibliography Extraction from Scientific Articles // Proceedings of the Second Workshop on Information Extraction from Scientific Publications. 2023. P. 101-106.
8. Ma A. et al. A deep-learning based citation count prediction model with paper metadata semantic features // Scientometrics. 2021. Vol. 126. No. 8. P. 6803-6823.
9. Lo K. et al. PaperMage: A Unified Toolkit for Processing, Representing, and Manipulating Visually-Rich Scientific Documents // Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2023. P. 495-507.
10. Po D. K. Similarity based information retrieval using Levenshtein distance algorithm // International Journal of Advances in Scientific Research and Engineering. 2020. Vol. 6. No. 04. P. 06-10.
11. Nurcahyawati V., Mustaffa Z. Online Media as a Price Monitor: Text Analysis using Text Extraction Technique and Jaro-Winkler Similarity Algorithm // 2020 Emerging Technology in Computing, Communication and Electronics (ETCCE). IEEE, 2020. P. 1-6.
12. Foppiano L. et al. Automatic extraction of materials and properties from superconductors scientific literature // Science and Technology of Advanced Materials: Methods. 2023. Vol. 3. No. 1. P. 2153633.
13. Petersen T. et al. Geo-quantities: A framework for automatic extraction of measurements and spatial context from scientific documents // Proceedings of the 17th International Symposium on Spatial and Temporal Databases. 2021. P. 166-169.
14. Chraibi A. et al. Extraction of measurements from medical reports // 10ème conférence Francophone en Gestion et Ingénierie des Systèmes Hospitaliers, GISEH2020. 2020.
15. Haviana S. F. C., Subroto I. M. I. Obtaining Reference’s Topic Congruity in Indonesian Publications using Machine Learning Approach // 2019 6th International Con-ference on Electrical Engineering, Computer Science and Informatics (EECSI). IEEE. 2019. P. 428-431.
16. Ermakova L. Bordignon F., Turenne N., Noel M. Is the Abstract a Mere Teaser? Evaluating generosity of article abstracts in the environmental sciences // Frontiers in Research Metrics and Analytics. 2018. Vol 3. P. 16.
17. El-Ebshihy A. et al. A platform for argumentative zoning annotation and scien-tific summarization // Proceedings of the 31st ACM International Conference on Infor-mation & Knowledge Management. 2022. P. 4843-4847.
18. Choi W. et al. Building an annotated corpus for automatic metadata extraction from multilingual journal article references // PloS one. 2023. Vol. 18. No. 1. P. E0280637.
19. Krause J. et al. Bootstrapping multilingual metadata extraction: a showcase in cyrillic // Proceedings of the Second Workshop on Scholarly Document Processing. 2021. P. 66-72.
20. Shapiro I., Saier T., Färber M. Sequence Labeling for Citation Field Extraction from Cyrillic Script References // Proceedings of the Workshop on Scientific Document Understanding; co-located with 36th AAAI Conference on Artificial Inteligence (AAAI 2022). 2022.
21. Indrawati A., Yoganingrum A., Yuwono P. Evaluating the quality of the indo-nesian scientific journal references using ParsCit, CERMINE and GROBID // Library Phi-losophy and Practice. 2019. P. 1-14.
22. Meuschke N. et al. A benchmark of pdf information extraction tools using a multi-task and multi-domain evaluation framework for academic documents // Interna-tional Conference on Information. Cham : Springer Nature Switzerland, 2023. P. 383-405.
23. Guo Z., Jin H. Reference metadata extraction from scientific papers // 2011 12th International Conference on Parallel and Distributed Computing, Applications and Technologies. IEEE. 2011. P. 45-49.
24. Beel J., Langer S., Genzmehr M., Muller C. Docear’s PDF inspector: title extraction from PDF files // Proceedings of the 13th ACM/IEEE-CS joint conference on Dig-ital libraries. New York, NY, USA: ACM, 2013. P. 443–444.
25. Jensen Z. et al. A machine learning approach to zeolite synthesis enabled by automatic literature data extraction // ACS central science. 2019. Vol. 5. No. 5. P. 892-899.
26. Färber M., Albers A., Schüber F. Identifying used methods and datasets in scientific publications // Proceedings of the Workshop on Scientific Document Under-standing co-located with 35th AAAI Conference on Artificial Inteligence (AAAI 2021). 2021.
27. Suryawati E., Widyantoro D. H. Combination of heuristic, rule-based and machine learning for bibliography extraction // 2017 5th International Conference on In-strumentation, Communications, Information Technology, and Biomedical Engineering (ICICI-BME). IEEE. 2017. P. 276-281.
28. Tkaczyk D. et al. CERMINE: automatic extraction of structured metadata from scientific literature // International Journal on Document Analysis and Recognition (IJDAR). 2015. Vol. 18. P. 317-335.
29. Romary L., Lopez P. Grobid-information extraction from scientific publica-tions // ERCIM News. 2015. Vol. 100.
30. Councill I. G., Giles C. L., Kan M. Y. ParsCit: an Open-source CRF Reference String Parsing Package // Proceedings of the 6th International Conference on Language Resources and Evaluation, LREC 2008. 2008. Vol. 8. P. 661-667.
31. Prasad A., Kaur M., Kan M. Y. Neural ParsCit: a deep learning-based reference string parser // International journal on digital libraries. 2018. Vol. 19. P. 323-337.
32. Constantin A., Pettifer S., Voronkov A. PDFX: fully-automated PDF-to-XML conversion of scientific literature // Proceedings of the 2013 ACM symposium on Doc-ument engineering. 2013. P. 177-180.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Presenting an article for publication in the Russian Digital Libraries Journal (RDLJ), the authors automatically give consent to grant a limited license to use the materials of the Kazan (Volga) Federal University (KFU) (of course, only if the article is accepted for publication). This means that KFU has the right to publish an article in the next issue of the journal (on the website or in printed form), as well as to reprint this article in the archives of RDLJ CDs or to include in a particular information system or database, produced by KFU.
All copyrighted materials are placed in RDLJ with the consent of the authors. In the event that any of the authors have objected to its publication of materials on this site, the material can be removed, subject to notification to the Editor in writing.
Documents published in RDLJ are protected by copyright and all rights are reserved by the authors. Authors independently monitor compliance with their rights to reproduce or translate their papers published in the journal. If the material is published in RDLJ, reprinted with permission by another publisher or translated into another language, a reference to the original publication.
By submitting an article for publication in RDLJ, authors should take into account that the publication on the Internet, on the one hand, provide unique opportunities for access to their content, but on the other hand, are a new form of information exchange in the global information society where authors and publishers is not always provided with protection against unauthorized copying or other use of materials protected by copyright.
RDLJ is copyrighted. When using materials from the log must indicate the URL: index.phtml page = elbib / rus / journal?. Any change, addition or editing of the author's text are not allowed. Copying individual fragments of articles from the journal is allowed for distribute, remix, adapt, and build upon article, even commercially, as long as they credit that article for the original creation.
Request for the right to reproduce or use any of the materials published in RDLJ should be addressed to the Editor-in-Chief A.M. Elizarov at the following address: amelizarov@gmail.com.
The publishers of RDLJ is not responsible for the view, set out in the published opinion articles.
We suggest the authors of articles downloaded from this page, sign it and send it to the journal publisher's address by e-mail scan copyright agreements on the transfer of non-exclusive rights to use the work.