• Main Navigation
  • Main Content
  • Sidebar

Russian Digital Libraries Journal

  • Home
  • About
    • About the Journal
    • Aims and Scopes
    • Themes
    • Editor-in-Chief
    • Editorial Team
    • Submissions
    • Open Access Statement
    • Privacy Statement
    • Contact
  • Current
  • Archives
  • Register
  • Login
  • Search
Published since 1998
ISSN 1562-5419
16+
Language
  • Русский
  • English

Search

Advanced filters

Search Results

LLM Confidence in Constructing Semantic Classifications of Arguments

Daniil Sergeevich Larionov, Elena Nikolaevna Nikitina, Ivan Valentinovich Smirnov
1155-1173
Abstract:

This article addresses the problem of quantifying the confidence of large language models (LLMs) in the automatic semantic classification of arguments with emotive predicates. Using Russian-language social media posts, we analyze verbs of fear (to scare, to be afraid, etc.) and emotional attitude (to like, to love) with the semantic roles of experiencer, causator, and object. The study aims to compare self-assessed confidence of LLM Claude Sonnet 4.5 with expert assessments of the model's reasoning texts in argument classification in the healthcare domain. The experiment utilized a stratified set of 300 examples using chain-of-thoughts reasoning in Russian and four-step confidence scale. The results showed a moderate Spearman correlation between the expert and model assessments. A statistically significant relationship was found only between self-assessment of the model and the actual classification accuracy, while expert assessment of the linguistic characteristics of reasoning was unrelated to accuracy. It was concluded that explicit LLM reasoning is not directly related to its self-assessment of confidence and is separate from the decision-making process; reasoning may be an important functional part of the user interface, but not of the research.

Keywords: semantic roles, argument classification, verb of emotion, large language models, LLM reasoning, LLM confidence.
1 - 1 of 1 items
Information
  • For Readers
  • For Authors
  • For Librarians
Make a Submission
Current Issue
  • Atom logo
  • RSS2 logo
  • RSS1 logo

Russian Digital Libraries Journal

ISSN 1562-5419

Information

  • About the Journal
  • Aims and Scopes
  • Themes
  • Author Guidelines
  • Submissions
  • Privacy Statement
  • Contact
  • eLIBRARY.RU
  • dblp computer science bibliography

Send a manuscript

Authors need to register with the journal prior to submitting or, if already registered, can simply log in and begin the five-step process.

Make a Submission
About this Publishing System

© 2015-2026 Kazan Federal University; Institute of the Information Society