ArXiv Introduces Schema Coherence Inventory: A New Tool for Assessing Belief Organization in Language Models
Summary:ArXiv introduces the Schema Coherence Inventory (SCI), a novel framework for evaluating whether language models organize assigned beliefs into their cognition or merely exhibit mood-driven behavior. The SCI employs 96 forced-choice items across eight levels of Beck's cognitive model, scoring responses based on length-normalized log-likelihood. Validation studies on five model backbones (1.7B-14B parameters) demonstrate SCI's effectiveness in distinguishing belief-driven behavior from mood-driven
Background and Motivation
In the field of artificial intelligence, language models are frequently used to simulate patients or affective agents. However, assessing whether these models truly organize assigned beliefs into their cognitive structure, rather than merely exhibiting mood-driven behavior, has remained a challenge. Existing evaluation methods lack specificity in measuring belief organization, often conflating emotional responses with cognitive structure.
Core Mechanism of Schema Coherence Inventory (SCI)
The SCI addresses this issue through the following mechanisms:
- Multi-level Assessment: Based on Beck's cognitive model, SCI employs eight cognitive levels to provide a systematic evaluation from superficial emotions to deep beliefs.
- Forced-choice Design: It includes 96 forced-choice items, each corresponding to three negative self-schemas and their positive counterparts, ensuring comprehensive and accurate assessment.
- Scoring Mechanism: The length-normalized log-likelihood score avoids biases from generation and parsing.
- Target Specificity Discrimination: By comparing responses under different belief conditions, SCI effectively distinguishes belief-driven behavior from mood-driven responses.
Experiments and Validation
The research team conducted six validation experiments covering five model backbones (1.7B-14B parameters), leading to the following conclusions:
- High Accuracy: Clinical raters achieved 79.5% accuracy in identifying model belief organization, significantly higher than the chance level (16.7%).
- Cross-model Effectiveness: SCI demonstrated target specificity across all model backbones, with an average improvement of 17.7 percentage points.
- Short Form Effectiveness: The 48-item short form correlated with the full form at r = 0.73, demonstrating its efficiency.
Clinical Application and Extension
The SCI is not only applicable to simulated patients and affective agents but also provides a new standardized tool for affective measurement and cognitive assessment in language models. The research team plans to extend the SCI to more task types, rater groups, languages, and cultural backgrounds to further validate its universality.
Conclusion
The release of the SCI provides a new technical path for affective and cognitive assessment of language models. By separating the influence of emotions and cognition, the SCI enables more accurate evaluation of model behavior patterns, offering AI researchers and developers a more reliable assessment tool.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-10-26)
Tags: #Language Models #Cognitive Assessment #Affective Measurement #ArXiv #AI Research
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments