• Main Navigation
  • Main Content
  • Sidebar

Russian Digital Libraries Journal

  • Home
  • About
    • About the Journal
    • Aims and Scopes
    • Themes
    • Editor-in-Chief
    • Editorial Team
    • Submissions
    • Open Access Statement
    • Privacy Statement
    • Contact
  • Current
  • Archives
  • Register
  • Login
  • Search
Published since 1998
ISSN 1562-5419
16+
Language
  • Русский
  • English

Search

Advanced filters

Search Results

Representation of Intraword Syntagmatic Relations in Vector Language Models

Daria Kirillovna Rodionova, Olga Aleksandrovna Mitrofanova
898-918
Abstract:

The paper discusses semantic structure representation of derivatives in language models, taking into account the intraword syntagmatic relations between derivational morphemes. Experiments were conducted using morphemic models developed by the Russian National Corpus (RNC), as well as fastText and ruRoBERTa models. The study is aimed at the verification of the hypothesis dealing with compositionality of derived words which are represented as aggregated morpheme vectors. In experiments we explore the representation of semantic relationships using fastText morpheme vectors and standard subword vectors in ruRoBERTa. The results indicate moderate sensitivity of fastText vectors to syntagmatic relations between morphemes as well as to derivational types. At the same time, it was found that aggregating morpheme vectors in fastText provides better representation of semantic relations between words compared to aggregating subword vectors in ruRoBERTa.


Standard BPE (Byte-Pair Encoding) and WordPiece tokenizers used in Transformer-based models are poorly interpretable with respect to linguistic data, as word segments do not always correspond to morphemes. The research problem lies in the need to assess the extent to which modern language models can capture linguistic features that characterize the relationships of derived words within word-formation families. The aim of the study is to evaluate the ability of predictive distributed vector embedding models to reproduce syntagmatic connections between morphemes within derived words and at the level of word-formation families in the Russian language.


The obtained results encourage the development of neural network architectures that take into account syntagmatic relations between morphemes, the improvement of morpheme tokenizers, and their integration into language models.

Keywords: language models, morphemic analysis, word-formation methods, compositionality.

LLM Applications to the task of Word Sense Disambiguation in Russian texts

Polina Andreevna Gousyatskaya, Natalia Valentinovna Loukachevitch
1118-1132
Abstract:

This study is dedicated to experiments in Word Sense Disambiguation on Russian-language material using generative (decoder-only) models of small and medium size. While the direct application of generative models is not an optimal approach for this task, such models demonstrate potential as semantic annotators of large volumes of raw data. The automation of semantic text annotation using generative models may help overcome the bottleneck of insufficient labeled data for training encoder-based models.


Previous studies show that state-of-the-art English and multilingual models can achieve accuracy above 90% on this task, while smaller models typically reach 80%+. The present study aims to determine whether a comparable task can be successfully addressed for small- and medium-scale models adapted to Russian, that do not require substantial computational resources.


The experiments reported in this paper are conducted both in a basic setting (one/few-shot prompting) with separate tests for narrow and broad context windows of the ambiguous lexeme, as well as under several modifications, such as context enrichment with paradigmatic information (e.g., hypernyms, hyponyms, domain labels of the target word) and ensemble approaches in which one model validates and refines the predictions of another. The study is based on the Russian annotated resource RuSemCor, annotated in terms of the RuWordNet semantic net.


The experiments ultimately pursue the development of an efficient and accessible pipeline for automatic semantic annotation of Russian texts, useful both for direct application and for preparing annotated data for machinelearning tasks.

Keywords: word sense disambiguation, LLM, computational semantics, classification.

Digital 3D-Objects Visualization in Forming Virtual Exhibitions

Nikolay Evgenvich Kalenov, Sergey Alexandrovich Kirillov, Irina Nikolaevna Sobolevskaya, Aleksandr Nikolaevich Sotnikov
418-432
Abstract: The paper is presents approaches to solving the problem of creating realistic interactive 3D web-collections of museum exhibits. The presentation of 3D-models of objects based on oriented polygonal structures is considered. The method of creating a virtual collection of 3D-models using interactive animation technology is described. It is also shown how a full-fledged 3D-model is constructed on the basis of individual exposure frames using photogrammetry methods. The paper assesses the computational complexity of constructing realistic 3D-models. For the creation of 3D-models in order to provide them to a wide range of users via the Internet, the so-called interactive animation technology is used. The paper presents the differences between the representations of full-fledged 3D-models and 3D-models presented in the form of interactive multiplication. The technology of creating 3D-models of objects from the funds of the State Biological Museum named K.A Timiryazev and the formation on their basis of the digital library “Scientific Heritage of Russia” of a virtual exhibition dedicated to the scientific activities of M.M. Gerasimov and his anthropological reconstructions, and vividly demonstrating the possibility of integrating information resources by means of an electronic library. The format of virtual exhibitions allows you to combine the resources of partners to provide a wide range of users with collections stored in museum, archival and library collections.
Keywords: photogrammetry, 3D-modeling, interactive animation, web-design, polygonal modeling.

Methodology, Results, and Comparative Analysis of the Use of Mask-RCNN and YOLO Neural Network Models for the Task of Automatically Selecting Mode Tracks on Ionograms of Oblique Ionospheric Radiosonde

Stepan Vadimovich Masyuchenko, Andrey Olegovich Schiriy
2540-2554
Abstract:

Automatic processing of ionospheric radiosonde ionograms is necessary both for operational monitoring of the ionosphere and for analyzing archived radiosonde data arrays. The central place in this task is occupied by automatic detection and classification of tracks – traces of reflected radio signal modes on ionograms. This paper presents the methodology, results, and comparative analysis of the use of neural network models (neural network architectures) for Mask RCNN and YOLO instance segmentation. The most successful models were the YOLO family, which provide both high accuracy and computational efficiency. At the same time, the exact pixel–by-pixel matching of the masks is not critical – the key requirement is the correct localization of the useful signal. The use of these models to identify tracks on ionograms has demonstrated moderate but sufficient efficiency: the main structures of the tracks are reliably identified. The most successful models were the YOLO family, which provide both high accuracy and computational efficiency. In the future, the set of models used will be expanded, primarily due to visual transformers.

Keywords: ionogram, signal detection, image segmentation, ionogram processing.

Analysis of Word Embeddings for Semantic Role Labeling of Russian Texts

Leysan Maratovna Kadermyatova, Elena Victorovna Tutubalina
1026-1043
Abstract: Currently, there are a huge number of works dedicated to semantic role labeling of English texts [1–3]. However, semantic role labeling of Russian texts was an unexplored area for many years due to the lack of train and test corpora. Semantic role labeling of Russian Texts was widely disseminated after the appearance of the FrameBank corpus [4]. In this approach, we analyzed the influence of the word embedding models on the quality of semantic role labeling of Russian texts. Micro- and macro- F1 scores on word2vec [5], fastText [6], ELMo [7] embedding models were calculated. The set of experiments have shown that fastText models averaged slightly better than word2vec models as applied to Russian FrameBank corpus. The higher micro- and macro- F1 scores were obtained on deep tokenized word representation model ELMo in relation to classical shallow embedding models.
Keywords: machine learning, ML-model, natural language processing, word embedding, semantic role labeling.

Analysing Machine Learning Models based on Explainable Artificial Intelligence Methods in Educational Analytics

Dmitriy Arturovich Minullin, Fail Mubarakovich Gafarov
294-315
Abstract:

The problem of predicting early dropout of students of Russian universities is urgent and therefore requires the development of new innovative approaches to solve it. To solve this problem, it is possible to develop predictive systems based on the use of student data, available in the information systems of universities. This paper investigates machine learning models for predicting early student dropout trained on the basis of student characteristics and performance data. The main scientific novelty of the work lies in the use of explainable AI methods to interpret and explain the performance of the trained machine learning models. The Explainable AI methods allow us to understand which of the input features (student characteristics) have the greatest influence on the results of the machine learning models. (student characteristics) have the greatest influence on the prediction results of trained models, and can also help to understand why the models make certain decisions. The findings expand the understanding of the influence of various factors on early dropout of students.

Keywords: educational analytics, data mining, machine learning, explainable AI.

Verb Forms in Emotion Recognition Tasks: Analysis of Data Characteristics and Classification Results

Polina Vladimirovna Iaroshenko, Anastasia Alexeevna Petrova, Natalia Valentinovna Loukachevitch
2240-2264
Abstract:

Emotion recognition in text represents an important task in natural language processing. This study investigates the influence of verb morphological form on emotion classification performance by encoder models and identifies factors that may determine morphological sensitivity in language models. Three models (ruBERT-tiny2, ruBERT-base, ruRoBERTa-large), fine-tuned on a combined corpus of four Russian-language datasets (22,852 examples), were employed in the experiment. Each model classified 5,676 morphological forms of 473 verbs from the Russian Emotion Lexicon. Results revealed systematic morphological influence: differences in classification performance between forms, as measured by F1-score, reached 20–25% across all models. To account for the observed patterns, training data characteristics were analysed. Frequency of morphological forms in the corpus was shown to partially correlate with classification performance, though it does not appear to be the sole determining factor. Analysis of emotion distribution across morphological forms revealed that in the training corpus, imperative forms occur predominantly in contexts associated with Joy. However, Sadness-labelled verbs in imperative mood are systematically classified by encoders as Anger. This discrepancy between training data characteristics and classification outcomes suggests that imperatives combined with sadness semantics are interpreted by models as expressions of aggression, indicating an interaction between grammatical form and verb semantics in determining emotion class.

Keywords: emotion classification, BERT, natural language processing, Russian language, morphology.

Constrained Decoding for Structural Error Suppression in Dependency Tree Generation with Large Language Models

Elena Denisovna Shamaeva
1269-1292
Abstract:

In the field of syntactic parsing, fine-tuning large language models to generate a sentence's syntactic structure in bracket form is a promising approach. Existing fine-tuned models demonstrate high values of syntax parsing metrics; however, in some cases they generate an incorrect bracket sequence (for example, with an unbalanced number of opening and closing parentheses). Therefore, this study aims to develop a method to reduce the number of incorrectly generated sequences. A system of constraints has been introduced on the set of tokens generated at each stage of the large language model. The implemented method was tested on four fine-tuned models (adapted and non-adapted with 4 and 8 billion parameters) on the SynTagRus dependency tree dataset. For all models, the number of incorrectly generated sequences decreased. However, some sentences were not parsed, because the LLMs generated an excessively long token sequence and were interrupted. The code for this study is publicly available.

Keywords: dependency parsing, large language model, fine-tuning, generation constraing, natural language processing.

Comparative Analysis of Geological Texts using Large Language Models

Michail Ivanovich Patuk, Vera Viktorovna Naumova
806-821
Abstract:

The rapid increase in the volume of publications in various fields of geology makes it crucial to introduce methods for automated processing of scientific texts. Large language models based on neural networks represent one of the most promising approaches to solving this challenge. The recent breakthroughs in artificial intelligence have made such models indispensable tools for researchers. Our work on semantic search for publications using additionally trained language models and measuring the similarity between geological texts yielded good results. However, the models we used were unable to perform in-depth text analysis. A comparative analysis of modern architectures identified the DeepSeek R1 model as belonging to a class of systems with advanced logical inference abilities. This type of model represents a fundamentally new level of quality in text generation. Based on the chosen model, we have developed a web service that provides unique functionality for comparative analysis of up to 5 scientific articles. The service supports multilingual sources, allowing users to input text in English, Chinese, Russian, etc. It generates structured reports in Russian, highlighting key theses, contradictions, and patterns. The proposed approach has been tested on geological publications, and the results have been promising.

Keywords: artificial intelligence, large language models, natural language processing, texts analysis, geology.

Exploring Post-Training Quantization of Large Language Models with a Focus on Russian Evaluation

Dmitrii Romanovich Poimanov, Mikhail Sergeevich Shutov
1138-1163
Abstract:

The rapid adoption of large language models (LLMs) has made quantization a central technique for enabling efficient deployment under real-world hardware and memory constraints. While English-centric evaluations of low-bit quantization are increasingly available, much less is known about its effects on morphologically rich and resource-diverse languages such as Russian. This gap is particularly important given the recent emergence of high-performing Russian and multilingual LLMs. In this work, we conduct a systematic study of 2-, 3-, and 4-bit post-training quantization (PTQ) for state-of-the-art Russian LLMs across different model scales (4B and 32B). Our experimental setup covers both standard uniform quantization and specialized low-bit formats, as well as lightweight finetuning for recovery in the most extreme 2-bit setting. Our findings highlight several important trends: (i) the tolerance of Russian LLMs to quantization differs across model families and scales; (ii) 4-bit quantization is generally robust, especially when advanced formats are used; (iii) 3-bit models expose sensitivity to calibration data and scaling strategies; and (iv) 2-bit models, while severely degraded under naive PTQ, can be partially restored through short finetuning. Empirical results show that the model's domain must be considered when using different quantization techniques.

Keywords: neural networks quantization, compression and optimization of large language models.

On the Applicability of Neural Networks in the Publishing Industry

Suhaylii Ilhom Shirinbegzoda, Daniil Andreevich Shishkin, Bogdan Sergeevich Usmanov, Nikolay Mikhailovich Borgest
960-975
Abstract:

The paper assesses the limits of applicability of large language models in editorial tasks within the publishing process and identifies the optimal format of interaction between humans and algorithmic systems.


The methodological basis of the study is a comparative experiment in which several popular neural network models — Alice AI, GigaChat, DeepSeek, Gemini, and ChatGPT — performed a statistical analysis of a control text in Russian. The quantitative characteristics of the text were determined: the number of words, characters with and without spaces, and the number of paragraphs. The obtained results were compared with reference values established using the MS Word text editor, which applies a deterministic character-counting algorithm.


The results of the experiment showed that neural network models demonstrate varying degrees of accuracy when performing tasks of quantitative text analysis. The main reason for such errors lies in the architecture of large language models and the use of tokenization algorithms, which break the direct connection between characters and the model’s internal representation of the text.


Based on the results obtained, the paper proposes the concept of a hybrid architecture for publishing information systems, in which generative language models are used to perform creative and analytical tasks, while operations requiring strict formal accuracy are assigned to specialized deterministic microservices. The proposed approach makes it possible to improve the reliability and predictability of intelligent publishing systems.

Keywords: аrtificial intelligence, publishing industry, large language models, neural networks, automation, tokenization, editorial workflow.

Intelligent Multimodal Neural Network Monitoring Service for the Surveillance Area

Razil Rustemovich Minneakhmetov
123-144
Abstract:

The article presents an approach to the development of an intelligent multimodal monitoring service for the surveillance area using large neural network models. The proposed solution is capable of analyzing heterogeneous data – video streams, environmental sensor signals (temperature, humidity, etc.), and event logs – to obtain a complete picture of what is happening. The main tools used are large language and visual models (for example, LLaMA, MiniCPM‑V, etc.) deployed locally using the Ollama platform, which provides autonomous and secure information processing without the need to transfer data to the cloud. A prototype system has been developed that works offline and is capable of detecting critical situations, abnormal deviations from the norm and contextually significant events in the observed area. The method of forming test scenarios and conducting a qualitative assessment of the model's performance using the metrics F1-measure, Precision, Recall on a set of various situations is described. The experimental results confirm the applicability of multimodal models for monitoring tasks: the prototype successfully recognizes complex patterns of behavior and demonstrates the potential of large models in building adaptive and scalable surveillance systems.

Keywords: intelligent service, multimodal monitoring, Ollama, Large Language Models, activity tracking, video analytics, artificial intelligence.

Using Open Archives of Scaled Vertical Radiosonding Ionograms as Labeled Data for Training Machine Learning Models

Andrey Olegovich Schiriy
532-545
Abstract:

The idea of using the available large arrays of ionogram processing results from vertical radiosonde of the ionosphere as training datasets for building predictive models using machine learning methods is put forward. The most common formats for saving the results of ionogram processing are considered, as well as some Internet resources with archives of freely available files of these formats. These datasets are used by us to build predictive models, including time series of critical frequencies of ionospheric layers. It is also possible to use some datasets of ionogram processing results to train models designed for automatic ionogram processing.

Keywords: ionosphere, radiosounding, vertical sounding of the ionosphere, ionogram, ionosphere characteristics, metadata.

Automatic Speech Recognition Quality Prediction based on Large Language Models

Anton Vyacheslavovich Polevoi
1189-1211
Abstract:

Despite the rapid development in the field of automatic speech recognition (ASR) systems, the recognition quality remains poor for audio recordings with acoustic degradation (music, crowd shouts, sounds of machinery, etc.). This becomes especially important when implementing models in critical areas that are characterized by increased attention to control and reliability of the results (aviation, medicine, autopilots, etc.). Error in such areas is very expensive and the use of models becomes impossible, even if the error occurs rarely. To reduce risks, it is advisable to evaluate the expected recognition quality in advance.


This article proposes an approach to predicting the Word Error Rate (WER) based on the acoustic characteristics of the signal and calculating the perplexity of language models. The proposed method involves the creation of diverse sets of audio data by applying various types of acoustic observations to pure speech samples at various levels of quality and intelligibility. Unlike previous studies, a complete set of speech features is extracted and analyzed: prediction of the value of the signal-to-noise ratio (SNR), neural network sound quality metrics (NISQA, etc.) as well as the perplexity of the text of the ASR hypothesis using the language model as an additional feature for training a unified model.


Experiments are being conducted using modern speech recognition architectures to demonstrate the effectiveness of the proposed method in predicting WER in various acoustic conditions. It is shown that the inclusion of perplexity significantly improves the quality of the WER prediction, in particular for data where acoustic features are weakly correlated with recognition errors. The results are applicable for automatic evaluation of the expected quality of speech recognition and filtering of audio inputs.

Keywords: WER prediction, acoustic degradation in speech recognition, perplexia, ASR confidence.

Large Language Models for Word-In-Context

Denis Vladislavovich Kokosinskii
1133-1154
Abstract:

Task-specific models have long dominated natural language processing tasks. However, general-purpose large language models (LLMs) have recently begun to successfully compete with highly specialized solutions across various NLP domains. In this paper, we investigate the applicability of LLMs to the task of estimating the semantic proximity of a word's meanings in a pair of usages, known as Word-in-Context (WiC). Drawing on the multilingual CoMeDi benchmark, we propose novel approaches to building automated WiC-systems based on LLMs. We conduct a systematic comparison of five different configurations in terms of quality and computational costs. In particular, we propose a configuration where LLM predictions are adjusted using a training set, without the need to fine-tune the LLM itself. Results on test sets across seven languages show that our proposed approaches enable LLMs to outperform all existing specialized systems, establishing a new state-of-the-art (SOTA) on the CoMeDi benchmark. Nevertheless, the achieved high quality comes with a significant increase in computational costs: LLM-based systems require several orders of magnitude more computations compared to compact specialized models (such as XL-DURel). This work represents a step towards understanding the trade-off between accuracy and resource efficiency when using modern LLMs in lexical semantics tasks.

Keywords: Word-in-Context, Large Language Model, Natural Language Processing.

Detection of Hallucinations Based on the Internal States of Large Language Models

Timur Rustemovich Aisin, Tatiana Vyacheslavovna Shamardina
1282-1305
Abstract:

In recent years, large language models (LLMs) have achieved substantial progress in natural language processing tasks and have become key instruments for addressing a wide range of applied and research problems. However, as their scale and capabilities grow, the issue of hallucinations — i.e., the generation of false, unreliable, or nonexistent information presented in a credible manner—has become increasingly acute. Consequently, analyzing the nature of hallucinations and developing methods for their detection has acquired both scientific and practical significance.


This study examines the phenomenon of hallucinations in large language models, reviews their existing classification, and investigates potential causes. Using the Flan-T5 model, we analyze differences in the model’s internal states when generating hallucinations versus correct responses. Based on these discrepancies, we propose two approaches for hallucination detection: one leveraging attention maps and the other utilizing the model’s hidden states. These methods are evaluated on data from HaluEval and Shroom 2024 benchmarks in tasks such as summarization, question answering, paraphrasing, machine translation, and definition generation. Additionally, we assess the transferability of the trained detectors across different hallucination types, in order to evaluate the robustness of the proposed methods.

Keywords: large language models, hallucinations, detection, Flan-T5, natural language processing, attention maps, hidden states, HaluEval, Shroom.

Application of Graph Neural Networks for Automatic Verification of BIM Models

Olga Vladimirovna Poletkina, Olga Muratovna Ataeva
1381-1398
Abstract:

Automating the verification of building information models for compliance with fire safety regulations remains a pressing issue in the architectural and construction industry. Existing automated verification systems rely on rule-based approaches that ignore the building's topological context and are poorly adapted to new projects. This paper examines the development and experimental verification of methods for predicting door fire protection parameters in BIM models using graph neural networks and validating the approach using real-world design data from seven residential buildings provided by a major real estate developer. A methodology for predicting door fire resistance classes based on relational graph convolutional networks is proposed, along with a pipeline for extracting data from a specialized format, constructing a graph, and generating features that take into account geometric, semantic, and topological characteristics. Experiments were conducted to predict the presence and fire resistance class with cross-project validation using the "one building out of sample" principle. The developed approach enables automated verification of fire protection parameters and reduces the time required to analyze building models. The use of graph neural networks ensures that topological context is taken into account and high prediction accuracy is achieved, while the use of real data from a major real estate developer confirms the practical applicability of the method.

Keywords: Building Information Modeling, graph neural networks, Relational Graph Convolutional Network, door fire resistance, Industry Foundation Classes, automated compliance checking, graph machine learning, cross-project validation.

Analysis of the Effectiveness of Subword Tokenizers in a Low-Resource Linguistic Environment: Implementation Experience for the Tajik Language

Mullosharaf Kurbonovich Arabov, Svetlana Sergeevna Khaybullina
546-564
Abstract:

This paper examines modern approaches to subword tokenization of texts as applied to the low-resource Tajik language, which is characterized by a complex morphological structure and a high degree of word-form variability. In the course of the study, a large-scale heterogeneous corpus was compiled and preprocessed, comprising 99 books and 134,497 textual articles of various genres and topics, with a total volume exceeding 33 million tokens. The corpus was cleaned of noise, normalized, and used as a basis for training and subsequent testing of subword models.


Based on this corpus, five tokenization models implementing the BPE, WordPiece, and Unigram algorithms were trained and analyzed using the Hugging Face Tokenizers and SentencePiece libraries. Comparative evaluation was conducted using a set of key metrics, including the proportion of out-of-vocabulary (OOV) words, the degree of text representation compression, tokenization speed, as well as characteristics of n-gram distribution, which make it possible to assess the ability of the models to capture the morphological and structural organization of the language. The experimental results made it possible to identify the strengths and weaknesses of different approaches to subword segmentation and to determine the most effective tokenization strategies under conditions of the morphological complexity of the Tajik language. The findings obtained can be used in the development of language models and applied NLP tools for Tajik and other low-resource languages, contributing to the expansion of their presence in the digital environment.

Keywords: Tajik language, subword tokenization, low-resource languages, BPE, WordPiece, Unigram, Hugging Face Tokenizers, SentencePiece, corpus linguistics, natural language processing (NLP).

Real-Time Generative Simulation of Game Environment

Eduard Sergeevich Bolshakov, Vlada Vladimirovna Kugurakova
188-212
Abstract:

This paper explores the potential of generative neural network simulations, focusing on the application of reinforcement learning methods and neural world models for creating interactive worlds. Key achievements in agent training using reinforcement learning are discussed. Special attention is given to neural world models, as well as generative models such as Oasis, DIAMOND, Genie, and GameNGen, which employ diffusion networks to generate realistic and interactive game worlds. The opportunities and limitations of generative simulation models are examined, including issues related to error accumulation and memory constraints, and their impact on the quality of generation. The conclusion presents suggestions for future research directions.

Keywords: video games, game environment, generative simulation, reinforcement learning, generative neural networks, gameplay simulation, world models.

Solving the Problem of Classifying the Emotional Tone of a Message with Determining the Most Appropriate Neural Network Architecture

Danis Ilmasovich Bagautdinov, Salman Salman, Vladislav Alekseevich Alekseev, Rustamdzhon Murodzhonovich Usmonov
396-413
Abstract:

To determine the most effective approach for solving the task of classifying the emotional tone of a message, we trained selected neural network models on various sets of training data. Next, based on the performance metric of the percentage of correctly classified responses on a test data set, we compared combinations of training data sets and various models trained on them. During the writing of this article, we trained four neural network models on three different sets of training data. By comparing the accuracy of the responses from each model trained on different training data sets, conclusions were drawn regarding the neural network model best suited for solving the task at hand.

Keywords: NLP, sentiment detection, neural networks, comparison of neural network models, LSTM, CNN, BiLSTM.

A Tool for Rapid Diagnostics of Memory in Neural Network Architectures of Language Models

Pavel Andreevich Gavrikov, Azamat Komiljon ugli Usmanov, Dmitriy Revayev, Sergey Nikolaevich Buzykanov
1346-1367
Abstract:

Large Language Models (LLMs) have evolved from simple n-gram systems to modern universal architectures; however, a key limitation remains the quadratic complexity of the self-attention mechanism with respect to input sequence length. This significantly increases memory consumption and computational costs, and with the emergence of tasks requiring extremely long contexts, creates the need for new architectural solutions. Since evaluating a proposed architecture typically requires long and expensive full-scale training, it is necessary to develop a tool that allows for a rapid preliminary assessment of a model’s internal memory capacity.


This paper presents a method for quantitative evaluation of the internal memory of neural network architectures based on synthetic tests that do not require large data corpora. Internal memory is defined as the amount of information a model can reproduce without direct access to its original inputs.


To validate the approach, a software framework was developed and tested on the GPT-2 and Mamba architectures. The experiments employed copy, inversion, and associative retrieval tasks. Comparison of prediction accuracy, error distribution, and computational cost enables a fast assessment of the efficiency and potential of various LLM architectures.

Keywords: large language models, neural network architecture, internal memory, long-term information retention, sequence processing, functional memory measurement, architecture comparison.

Automated Students' Short Answers Grading using Language Models

Chulpan Bakievna Minnegalieva, Ilnur Ilhamovich Kashapov, Olga Dmitrievna Morozova
278-293
Abstract:

Methods for assessing student answers using language models are currently being studied by various specialists. The results of automated assessment depend on the subject area and characteristics of the academic discipline. This paper analyzes the students’ answers received during the course «Computer Graphics and Design». It is proposed to determine the cosine similarity of document vectors obtained using language models and refine the estimates by checking keywords. The results obtained can be used for preliminary assessment of students' answers and are the basis for further research.

Keywords: language model, knowledge control, natural text processing, student answer keyword, automated short answer grading, cosine similarity, document vector representation, BERT, word2vec, open-ended question.

Integration of Semantic and Mathematical Modeling for the Analysis of Energy Security Problems

Alexey Genadevich Massel, Timur Gabilovich Mamedov
842-859
Abstract:

The study addresses the problem of integrating cognitive and mathematical modeling in research on the development directions of the fuel and energy complex, taking into account energy security requirements. The relevance of the work is due to the fact that in the existing two-level research methodology, the transition from the results of qualitative analysis using cognitive modeling to the parameters of the mathematical model is largely performed manually, which reduces the reproducibility of numerical experiments and limits the efficiency of accumulated knowledge usage. The aim of the work is to develop a software component that ensures the combined use of cognitive and mathematical models within an Energy Knowledge Ecosystem. A software component is proposed, implemented as part of the INTEC‑SAW suite, which provides the transformation of changes in the cognitive model into the parameters of the economic-mathematical model, as well as the reverse interpretation of calculation results. Technology for conducting numerical experiments has been developed, including the construction of semantic (ontological and cognitive) models, formation of computational scenarios, execution of optimization calculations, and presentation of results, distinguished by the automation of the joint use of ontological, cognitive, and economic-mathematical models. To account for uncertainty, a numerical method of stochastic parameter adjustment based on cognitive weights is proposed. The effectiveness of the approach is demonstrated through a numerical experiment investigating the impact of CO₂ emission constraints on the energy balances of the Siberian Federal District. The practical significance of the work lies in increasing the validity and reproducibility of research on the development of the fuel and energy complex through the coordinated use of qualitative and quantitative analysis tools.

Keywords: energy complex, energy security, cognitive modeling, ontologies, computational experiment, linear programming.

RusHallu-RAG: a Comprehensive Approach to Hallucination Detection in Russian-Language RAG Systems

Fedor Alekseevich Sadkovskii, Regina Ruslanovna Nasyrova, Alexey Andreevich Sorokin
2188-2214
Abstract:

This paper introduces RusHallu-RAG, the first comprehensive benchmark for detecting hallucinations in Russian-language RAG systems. The dataset of 1,000 query-answer pairs covers general knowledge and scientific domains with controlled context relevance balancing. A fine-grained taxonomy of six hallucination types is proposed and used for annotation at both response and span levels with human experts and LLMs.


For training a hallucination detector, a balanced dataset of 3,000 examples is collected using synthetic generation methods. A detector based on YandexGPT-5-Lite-8B is trained via SFT. We evaluate 15 open-weight and proprietary models alongside the trained detector. Results show that model size does not guarantee performance: medium-sized models (20–33B) sometimes outperform larger ones. The trained detector achieves an average accuracy of 63.7%, ranking second after proprietary Gemini-2.5-Pro, with a gain of 47.2 p.p. over the baseline. The gap between proprietary and open-weight models is reduced from 20–35 to 8.4 p.p., confirming the effectiveness of the approach. The benchmark, code, and trained model are publicly released.

Keywords: hallucination detection, retrieval-augmented generation (RAG), large language models (LLMs), benchmarking, russian language, natural language processing (NLP), question answering (QA), model evaluation, supervised fine-tuning (SFT), hallucination taxonomy.

Method of Pre-Assessment of Students' Answers Based on the Vector Model of Documents

Chulpan Bakievna Minnegalieva, Gulshat Alfisovna Sabitova, Almaz Maratovich Gayaliev
324-339
Abstract:

This article discusses the application of vector models for the preliminary analysis of students' free-form answers. Vector representations of words and documents were obtained using word2vec, doc2vec, and BERT models. The similarity between the answer given by the student and the correct answer was determined using the cosine measure. It was found that vector models allow identifying obviously incorrect answers with sufficient accuracy. For answers that are close in wording, an additional verification step is proposed. Using word2vec, binary classification of answers to certain questions was performed, and accuracy, precision, recall and F1-measure estimates were given.

Keywords: vector model, word2vec, doc2vec, BERT, cosine similarity, vector representation.
1 - 25 of 128 items 1 2 3 4 5 6 > >> 
Information
  • For Readers
  • For Authors
  • For Librarians
Make a Submission
Current Issue
  • Atom logo
  • RSS2 logo
  • RSS1 logo

Russian Digital Libraries Journal

ISSN 1562-5419

Information

  • About the Journal
  • Aims and Scopes
  • Themes
  • Author Guidelines
  • Submissions
  • Privacy Statement
  • Contact
  • eLIBRARY.RU
  • dblp computer science bibliography

Send a manuscript

Authors need to register with the journal prior to submitting or, if already registered, can simply log in and begin the five-step process.

Make a Submission
About this Publishing System

© 2015-2026 Kazan Federal University; Institute of the Information Society