LLMs Show High Consistency but Uneven Reliability in Veteran Crisis Annotation

Summary:A study evaluated LLM-based annotation of crisis severity in veteran-related social media across 149 posts, five severity levels, 25 open-source and proprietary models, eight prompt variants, and annotations from eight veteran subject-matter experts. The strongest models achieved an inter-model ordinal agreement of 0.8336, above the experts' nominal agreement of 0.494. However, this consistency did not reliably translate into expert alignment or detection of rare, high-severity suicidal-crisis c

---

Core Finding

This study used LLMs to annotate crisis severity in veteran-related social media posts and asked a question that can be obscured by the headline metric: does agreement among models actually reproduce the judgments of veteran subject-matter experts?\n The evaluation covered 149 posts, five severity levels, 15 open-source models, 10 proprietary models, eight prompt variants, and annotations from eight veteran experts. Inter-model ordinal agreement reached 0.8336, while nominal agreement among experts was 0.494. The difference requires care: expert disagreement often occurred between adjacent severity levels, so exact-label agreement naturally penalizes boundary uncertainty. It does not establish that the models are more reliable than human experts.

More importantly, higher model consistency did not reliably translate into closer agreement with experts or better detection of rare, high-severity suicidal-crisis cases. A stable set of models can make the same mistake, especially when aggregate accuracy is dominated by low-risk examples.

How the Evaluation Separates Reliability From Consistency

The study separates several properties that are often conflated:

• Agreement with expert annotations: how closely model labels track human judgments;
• Inter-model agreement: whether different models and prompts produce relatively consistent rankings;
• Prompt sensitivity: the effect of baseline prompts, demonstrations, and prompt variants;
• High-severity detection: performance on rare acute-crisis cases involving suicidal ideation;
• Demonstration contamination: susceptibility to flipped labels and the effect of explanatory demonstrations.

This separation matters in safety-critical classification. A reproducible label is not necessarily a correct label, particularly when rare high-risk examples are evaluated through macro-averaged metrics.

Why Consistency Does Not Equal Expert Alignment

Crisis severity is an ordinal classification problem. If experts disagree mainly between neighboring levels, nominal agreement understates their semantic proximity. Ordinal agreement is therefore more informative than exact-match agreement, but it still does not solve the deployment problem.

A model may consistently place medium-risk cases in a low-risk band while still showing high inter-model agreement. Production evaluation should therefore report:

1. Per-class precision, recall, and confusion matrices;
2. The proportion of adjacent-label errors versus cross-level errors;
3. Recall and miss rates for high-severity cases;
4. Whether the model reproduces genuine expert uncertainty or merely removes it;
5. Confidence intervals under prompt, model, and decoding changes.

Recall for rare high-risk cases is especially important. Overall accuracy can remain high while a clinically significant fraction of suicidal-crisis posts is missed.

Demonstrations Become Part of the Attack Surface

Label-flipped experiments showed model-specific sensitivity to misleading in-context examples. Explanatory demonstrations, by contrast, did not consistently improve performance. Supplying more explanatory text or more demonstrations is therefore not equivalent to adding reliable supervision.

During in-context learning, models can learn decision shortcuts from supplied examples. A flipped label may become part of the apparent task mapping, while explanation-heavy prompts can add context length and noise without adding verifiable evidence. Production systems should review demonstrations independently, test their influence under perturbation, and avoid treating any single example as ground truth.

Model Differences and Selection

Proprietary models generally achieved better expert alignment and stability in this experiment. GPT-5.6-Terra had the best overall performance, while Qwen2.5-7B was strongest for identifying acute-crisis cases with suicidal ideation. These results are conditional on the dataset, prompts, decoding settings, and scoring protocol; the abstract does not provide complete per-model metrics or independent replication.

The dataset is also small for a safety-critical task. With only 149 posts and a limited number of severe cases, a handful of examples can materially change recall estimates. Sarcasm, denial, ambiguous help-seeking, and platform-specific language further complicate annotation. A larger external validation set should cover time periods, platforms, languages, and risk strata.

Deployment Guidance

The findings support a constrained workflow rather than autonomous triage:

• Use LLMs for bulk pre-annotation and prioritization;
• Require expert review for high-severity, borderline, and model-uncertain cases;
• Keep the existing human crisis-response pathway authoritative;
• Log model versions, prompts, evidence, and human corrections;
• Re-run a high-risk replay set after any model, prompt, or decoding change;
• Monitor class-specific drift, rare-event recall, and demonstration contamination separately.

The central engineering lesson is that model consistency and judgment reliability are different properties. LLMs can expand the coverage of expert annotation, but expert standards, calibration, and explicit escalation paths remain essential in high-stakes mental-health applications.

---

Source: ArXiv AI (cs.AI) (2026-11-10)

— END —

Tags: #LLM Evaluation Follow #Mental Health Follow #Crisis Severity Follow #Prompt Sensitivity Follow #Human-AI Agreement Follow