Ontological Approach to Knowledge Graph Assessment in the Domain of Mechanical Product Lifecycle Management Systems
Main Article Content
Abstract
This paper examines the application of an ontological approach to constructing a dataset for evaluating and comparing context enrichment systems for large language models using knowledge graphs in the domain of mechanical product lifecycle management systems. In this domain, obtaining the required amount of textual data with a formal logical structure to form an evaluation set without using generated synthetic data is challenging. To avoid introducing distortions and hallucinations when generating evaluation data, a novel solution to the data deficiency is proposed. This solution involves extracting ontology directly from product and assembly files compliant with the STandard for Exchange of Product Model Data. This potentially enables the use of all product data as a source for scaling evaluation data. The goal of this paper is to create a dataset of structured textual data in the domain of mechanical product lifecycle management systems, develop an evaluation methodology, and implement context enrichment pipelines for large language models with and without knowledge graphs to analyze the contribution of data-structure-extracting systems to the quality of generated responses. In this paper: a new source of evaluation data is proposed, a new methodology for generating text evaluation data while preserving the logical structure is developed, a pipeline for using the generated evaluation data is implemented, and evaluation results are obtained that confirm the positive contribution of systems with the extraction of structured data to the quality of generated responses in the domain of mechanical product lifecycle management systems.
Article Details
References
2. Xiang Z. et al. When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation // arXiv: 2506.05690. https://doi.org/10.48550/arXiv.2506.05690
3. Han H. et al. Rag vs. graphrag: A systematic Evaluation and Key Insights // arXiv: 2502.11371. 2025. https://doi.org/10.48550/arXiv.2502.11371
4. Han H. et al. Retrieval-Augmented Generation with Graphs (GraphRAG) // arXiv: 2501.00309. 2024. https://doi.org/10.48550/arXiv.2501.00309
5. Gajderowicz B., Bhardwaj A., Fox M. RAG and Ontologies for Information Retrieval: A Literature Review. 2025. URL: https://eil.mie.utoronto.ca/wp-content/uploads/2025/09/ONTOLLM_2025_Aug20v11.pdf
6. Huang Y., Fung T.Y., DeLaurentis D.A. Addressing Complexity in System of Systems With GraphRAG: An AI‐Driven Framework for Dynamic Data Integration // Systems Engineering. 2025. e70012. https://doi.org/10.1002/sys.70012
7. da Cruz T., Tavares B., Belo F. Ontology Learning and Knowledge Graph Construction: A Comparison of Approaches and Their Impact on RAG Performance // arXiv: 2511.05991. 2025. https://doi.org/10.48550/arXiv.2511.05991
8. GOST R 56862-2016. Nacional'nyj standart Rossijskoj Federacii “Sistema upravlenia ziznennym ciklom. Razrabotka koncepcii izdelia i tehnologij. Terminy i opredelenia”. URL: https://protect.gost.ru/document.aspx?control=7&id=202875 (last access: 12.02.2026)
9. GOST R ISO 10303-1-2022. Nacional'nyj standart Rossijskoj Federacii “Sistemy avtomatizacii proizvodstva i ih integracia. Predstavlenie dannyh ob izdelii i obmen etimi dannymi. Cast' 1. Obsie predstavlenia i osnovopolagausie principy (ISO 10303-1:2021, IDT)” URL: https://meganorm.ru/Data/792/79232.pdf
10. Kwon S., Monnier L.V., Barbau R., Bernstein W.Z. A New Implementation of OntoSTEP: Flexible Generation of Ontology and Knowledge Graphs of EXPRESS-Driven Data // ASME Journal of Computing and Information Science in Engineering. 2022. Vol. 22 (2). 024502. https://doi.org/10.1115/1.4053079
11. Stepcode. URL: https://github.com/stepcode/stepcode/tree/develop/ data/ ap214e3
12. Zhang Q. et al. A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models // arXiv: 2501.13958. 2025.https://doi.org/10.48550/arXiv.2501.13958
13. Fuchs N.E., Kaljurand K., Kuhn T. Attempto controlled english for knowledge representation //Reasoning Web: 4th International Summer School 2008, Venice, Italy, September 7–11, 2008, Tutorial Lectures. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008. P. 104–124. https://doi.org/10.1007/978-3-540-85658-0
14. Zaitoun A., Sagi T., Peleg M. Generating ontology-learning training-data through verbalization //Proc. of the AAAI Symposium Series. 2024. Vol. 4. No.1. P. 233–241. https://doi.org/10.1609/aaaiss.v4i1.31797
15. Fuchs N.E. et al. Attempto controlled english: A knowledge representation language readable by humans and machines // Reasoning Web: First International Summer School 2005. Springer Berlin Heidelberg, 2005. P. 213–250. https://doi.org/10.1007/11526988_6
16. Dong S. et al. Knowledge-Graph Based RAG System Evaluation Framework // arXiv: 2510.02549. 2025. https://doi.org/10.48550/arXiv.2510.02549
17. Chen E. et al. Comparing RAG and GraphRAG for Page-Level Retrieval Question Answering on Math Textbook // arXiv: 2509.16780. 2025. https://doi.org/10.48550/arXiv.2501.13958
18. Xiao Y. et al. GraphRAG-Bench: Challenging Domain-Specific Reasoning for Evaluating Graph Retrieval-Augmented Generation // arXiv: 2506.02404. 2025. https://doi.org/10.48550/arXiv.2506.02404
19. Es S. et al. Ragas: Automated evaluation of retrieval augmented generation // Proc. of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. 2024. P. 150–158. https://doi.org/10.48550/arXiv.2309.15217
This work is licensed under a Creative Commons Attribution 4.0 International License.
Presenting an article for publication in the Russian Digital Libraries Journal (RDLJ), the authors automatically give consent to grant a limited license to use the materials of the Kazan (Volga) Federal University (KFU) (of course, only if the article is accepted for publication). This means that KFU has the right to publish an article in the next issue of the journal (on the website or in printed form), as well as to reprint this article in the archives of RDLJ CDs or to include in a particular information system or database, produced by KFU.
All copyrighted materials are placed in RDLJ with the consent of the authors. In the event that any of the authors have objected to its publication of materials on this site, the material can be removed, subject to notification to the Editor in writing.
Documents published in RDLJ are protected by copyright and all rights are reserved by the authors. Authors independently monitor compliance with their rights to reproduce or translate their papers published in the journal. If the material is published in RDLJ, reprinted with permission by another publisher or translated into another language, a reference to the original publication.
By submitting an article for publication in RDLJ, authors should take into account that the publication on the Internet, on the one hand, provide unique opportunities for access to their content, but on the other hand, are a new form of information exchange in the global information society where authors and publishers is not always provided with protection against unauthorized copying or other use of materials protected by copyright.
RDLJ is copyrighted. When using materials from the log must indicate the URL: index.phtml page = elbib / rus / journal?. Any change, addition or editing of the author's text are not allowed. Copying individual fragments of articles from the journal is allowed for distribute, remix, adapt, and build upon article, even commercially, as long as they credit that article for the original creation.
Request for the right to reproduce or use any of the materials published in RDLJ should be addressed to the Editor-in-Chief A.M. Elizarov at the following address: amelizarov@gmail.com.
The publishers of RDLJ is not responsible for the view, set out in the published opinion articles.
We suggest the authors of articles downloaded from this page, sign it and send it to the journal publisher's address by e-mail scan copyright agreements on the transfer of non-exclusive rights to use the work.