Constrained Decoding for Structural Error Suppression in Dependency Tree Generation with Large Language Models
Main Article Content
Abstract
In the field of syntactic parsing, fine-tuning large language models to generate a sentence's syntactic structure in bracket form is a promising approach. Existing fine-tuned models demonstrate high values of syntax parsing metrics; however, in some cases they generate an incorrect bracket sequence (for example, with an unbalanced number of opening and closing parentheses). Therefore, this study aims to develop a method to reduce the number of incorrectly generated sequences. A system of constraints has been introduced on the set of tokens generated at each stage of the large language model. The implemented method was tested on four fine-tuned models (adapted and non-adapted with 4 and 8 billion parameters) on the SynTagRus dependency tree dataset. For all models, the number of incorrectly generated sequences decreased. However, some sentences were not parsed, because the LLMs generated an excessively long token sequence and were interrupted. The code for this study is publicly available.
Article Details
References
2. Vasiliev S., Korobkin D., Fomenkov S. Extracting the Component Composition Data of Inventions from Russian Patents using Dependency Tree Analysis // 2023 International Conference on Industrial Engineering, Applications and Manufacturing (ICIEAM), Sochi. 2023. P. 1030–1034. https://doi.org/10.1109/ICIEAM57311.2023.10139170
3. Alonso M. A., Gómez-Rodríguez C., Vilares J. On the use of parsing for named entity recognition // Applied sciences. 2021. Vol. 11 (3). 1090. https://doi.org/10.3390/app11031090
4. Nikolaev I.E. Knowledge and skills extraction from the job requirements texts // Ontology of Designing. 2023. Vol. 13 (2). P. 282–293. https://doi.org/10.18287/2223-9537-2023-13-2-282-293
5. Morozov D. et al. Exploring the Feature Space for Cross-Domain Assessing the Complexity of Russian-Language Texts // 2024 Ivannikov ISPRAS Open Conference, Moscow. 2024. P. 1–8. https://doi.org/10.1109/ISPRAS64596.2024.10899137
6. Liu T., Sun Y., Wu J. et al. Unsupervised Paraphrasing under Syntax Knowledge // Proc. of the AAAI Conference on Artificial Intelligence. 2023. P. 13273–13281. https://doi.org/10.1609/aaai.v37i11.26558
7. Taufiq U., Pulungan R., Suyanto Y. Named entity recognition and dependency parsing for better concept extraction in summary obfuscation detection // Expert Systems with Applications. 2023. Vol. 217. 119579. https://doi.org/10.1016/j.eswa.2023.119579
8. Chen D., Manning C.D. A fast and accurate dependency parser using neural networks // Proc. of the Conference on Empirical Methods in Natural Language Processing, Doha, Qatar. 2014. P. 740–750. https://doi.org/10.1016/j.eswa.2023.11957910.3115/v1/D14-1082
9. Andor D., Alberti C., Weiss D. et al. Globally normalized transition-based neural networks // Proc. of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), Berlin, Germany. 2016. P. 2442–2452. https://doi.org/10.1016/j.eswa.2023.11957910.18653/v1/P16-123
10. Dozat T., Manning C.D. Deep Biaffine Attention for Neural Dependency Parsing // International Conference on Learning Representations. 2017.
11. Ezquerro A., Gómez-Rodríguez C., Vilares D. Better Benchmarking LLMs for Zero-Shot Dependency Parsing // Proc. of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), Tallinn, Estonia. 2025. P. 121–135.
12. Hromei C. D., Croce D., Basili R. U-DepPLLaMA: Universal Dependency Parsing via Auto-regressive Large Language Models // IJCoL. Italian Journal of Computational Linguistics. 2024. Vol. 10 (1). https://doi.org/10.1016/j.eswa.2023.11957910.4000/125nm
13. Lin B. et al. ChatGPT is a potential zero-shot dependency parser // arXiv:2310.16654. https://doi.org/10.48550/arXiv.2310.16654
14. Matsuda H., Ma C., Asahara M. Step-by-step Instructions and a Simple Tabular Output Format Improve the Dependency Parsing Accuracy of LLMs // Proc. of the 18th International Conference on Parsing Technologies (IWPT, SyntaxFest 2025), Ljubljana, Slovenia. Association for Computational Linguistics: 2025. P. 11–19.
15. Zhou H., Chersoni E., Hsu Y. Y. Branching Out: Exploration of Chinese Dependency Parsing with Fine-tuned Large Language Models // Conference on Recent Advances in Natural Language Processing (RANLP 2025), Varna, Bulgaria. INCOMA Ltd: 2025. P. 1437–1445.
16. Tian Y., Xia F., Song Y. Large language models are no longer shallow parsers // Proc. of the 62nd Annual Meeting of the Association for Computational Linguistics, Thailand. Association for Computational Linguistics: 2024. P. 7131–7142.
17. Bai X., Wu J., Chen Y., Wang Z., Zhang Y. Constituency parsing using llms // IEEE Transactions on Audio, Speech and Language Processing IEEE. 2025. Vol. 33. P. 3762–3775. https://doi.org/10.1109/TASLPRO.2025.3600867
18. Ginn M., Palmer A. LLM Dependency Parsing with In-Context Rules // Proc. of the 1st Joint Workshop on Large Language Models and Structure Modeling, Vienna, Austria. Association for Computational Linguistics. 2025. P. 189–196.
19. Kim T. Revisiting the practical effectiveness of constituency parse extraction from pretrained language models // Proc. of the 29th International Conference on Computational Linguistics, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. 2022. P. 5398–5408.
20. Sun X. et al. Pushing the Limits of ChatGPT on NLP Tasks // arXiv: 2306.09719. 2023. https://doi.org/10.48550/arXiv.2306.09719
21. Zeman D., Hajic J., Popel M. et al.CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies // Proc. of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, Vancouver, Canada. 2017. P. 1–19. https://doi.org/10.18653/v1/K17-3001
22. Tikhomirov M., Chernyshev D. Facilitating large language model Russian adaptation with learned embedding propagation // Journal of Language and Education. 2024. Vol. 10, No.4 (40). P. 130–145. https://doi.org/10.17323/jle.2024.22224
23. Shamaeva E.D. Sravnenie neirosetevih sintaksicheskih analyzatorov dlya ruskogo yazika // Komp'uternaya lingvistika i vichislitelnie ontologii. 2025. No. 1. P. 26–47. https://doi.org/10.17586/3033-5582-2025-9-26-47
This work is licensed under a Creative Commons Attribution 4.0 International License.
Presenting an article for publication in the Russian Digital Libraries Journal (RDLJ), the authors automatically give consent to grant a limited license to use the materials of the Kazan (Volga) Federal University (KFU) (of course, only if the article is accepted for publication). This means that KFU has the right to publish an article in the next issue of the journal (on the website or in printed form), as well as to reprint this article in the archives of RDLJ CDs or to include in a particular information system or database, produced by KFU.
All copyrighted materials are placed in RDLJ with the consent of the authors. In the event that any of the authors have objected to its publication of materials on this site, the material can be removed, subject to notification to the Editor in writing.
Documents published in RDLJ are protected by copyright and all rights are reserved by the authors. Authors independently monitor compliance with their rights to reproduce or translate their papers published in the journal. If the material is published in RDLJ, reprinted with permission by another publisher or translated into another language, a reference to the original publication.
By submitting an article for publication in RDLJ, authors should take into account that the publication on the Internet, on the one hand, provide unique opportunities for access to their content, but on the other hand, are a new form of information exchange in the global information society where authors and publishers is not always provided with protection against unauthorized copying or other use of materials protected by copyright.
RDLJ is copyrighted. When using materials from the log must indicate the URL: index.phtml page = elbib / rus / journal?. Any change, addition or editing of the author's text are not allowed. Copying individual fragments of articles from the journal is allowed for distribute, remix, adapt, and build upon article, even commercially, as long as they credit that article for the original creation.
Request for the right to reproduce or use any of the materials published in RDLJ should be addressed to the Editor-in-Chief A.M. Elizarov at the following address: amelizarov@gmail.com.
The publishers of RDLJ is not responsible for the view, set out in the published opinion articles.
We suggest the authors of articles downloaded from this page, sign it and send it to the journal publisher's address by e-mail scan copyright agreements on the transfer of non-exclusive rights to use the work.