FairSumEval Separates Evidence Omissions from Unsupported Agreement in Discharge Summaries
Summary:FairSumEval introduces an evidence-aware, fact-level evaluation framework for clinical discharge summaries. It maps reference and generated facts to the available clinical evidence, then separates matches, omissions, and additions. Its complementary metrics, Soft IoU-S and Soft IoU-NS, measure agreement for facts with and without identified evidence support. Across 200 MIMIC-III admissions, only 18.8% of physician-written reference facts had identifiable support in the prior evidence, versus 65
Core Finding: A Reference Summary Is Not the Only Ground Truth
FairSumEval addresses a fundamental evaluation mismatch in discharge-summary generation. A physician-written reference does more than restate the admission record: it may contain clinical synthesis, handoff information, and decisions made at discharge. A model's failure to reproduce facts that were never present in the input is not necessarily an omission. Conversely, high overlap with the reference does not prove that the generated summary is clinically grounded.
Reference-only metrics such as ROUGE, BERTScore, or an LLM judge can conflate several distinct outcomes:
- Recoverable evidence omitted by the model: a genuine omission;
- Reference-only information unsupported by the input: not something the model can reliably reproduce;
- Unsupported facts introduced by the model: hallucinations or unsupported additions;
- Shared unsupported content: accidental agreement rather than reliable summarization.
FairSumEval therefore moves evaluation to the fact level and uses the available clinical evidence as a third coordinate.
Framework and Metrics
The method decomposes reference and generated summaries into atomic facts, checks whether each fact is supported by the model-visible clinical record, and constructs a fact-level outcome map:
- matched facts shared by the reference and generation;
- omitted reference facts that are supported by the evidence;
- added generation facts absent from the reference;
- facts for which no support is identified in the input evidence.
Two complementary metrics capture agreement in these different regions:
- Soft IoU-S measures agreement for facts with identified evidence support;
- Soft IoU-NS measures agreement for facts without identified evidence support.
The soft formulation allows semantic rather than exact lexical matches. The important result is the separation of two diagnostic questions: whether the model recovers supported clinical facts, and whether it expands beyond the evidence in ways that resemble the reference.
IoU remains sensitive to set size. A summary can contain many correct matches but still receive a lower score when omissions or additions are numerous. The metrics should therefore be interpreted alongside fact counts, evidence recall, and the severity of newly introduced claims.
Evidence Support Rates Reverse the Naive Ranking
The evaluation covers 200 MIMIC-III admissions. Only 18.8% of physician reference facts had identifiable support in the prior clinical evidence. By contrast, 65.4% and 63.5% of facts generated by a general-purpose and a medically specialized model had identified support.
This does not mean that the models are more accurate than physicians. It means that the references contain substantial clinical synthesis and discharge-time decisions that cannot be reconstructed from the model input. Treating all reference-only information as mandatory target content can penalize useful evidence extraction while rewarding unsupported agreement.
Overlap was also substantially higher for supported content, at 0.255 and 0.236, than for unsupported content, at 0.030 and 0.026. Conventional similarity scores are therefore dominated by agreement on content whose grounding cannot be established from the input.
Why One Score Is Not Enough
The two proposed scores correlate only weakly across admissions, with ρ=0.22 and ρ=0.17. A case may show strong recovery of supported facts while also containing many unsupported expansions, and the opposite pattern is possible as well.
The underlying issue is task definition, not model scale. Discharge-summary generation combines evidence extraction, handling of unavailable information, and the expression of discharge decisions. Treating it as a single text-compression objective makes conventional metrics structurally incomplete.
The study has several limitations. MIMIC-III represents one dataset and one documentation environment, and the sample of 200 admissions is sufficient for methodological validation but not for broad claims about hospital-level generalization. The abstract also does not report agreement or error rates for the fact decomposition and evidence-linking stages, which are likely to influence the final scores.
Practical Recommendations
Medical summarization pipelines should not collapse evaluation into one aggregate similarity score. They should track at least:
- evidence-grounded fact coverage;
- fact-level precision;
- unsupported addition rate;
- Soft IoU-S and Soft IoU-NS as separate diagnostics;
- severity-weighted errors for medications, diagnoses, allergies, and procedures.
Training can use the same separation. Extract verifiable facts from source records, supervise the summary to reproduce those facts, and explicitly mark uncertainty when the input does not determine a conclusion. An LLM judge should receive both the reference and source evidence and return a support decision for every fact. Without source evidence, it will likely collapse into another reference-similarity metric.
For production use, FairSumEval is best treated as an offline model-selection and error-analysis layer. Prospective deployment still requires physician review, cross-institution validation, and strict precision requirements for high-risk clinical facts.
Engineering Value
FairSumEval does not improve the generator directly. Instead, it corrects the evaluation objective for a task where the reference contains information unavailable from the input. By separating recoverable omissions from unsupported agreement and expansion, it offers a more actionable diagnostic interface for clinical summarization and other generation settings with incomplete source evidence.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-11-13)
Tags: #Medical NLP #Clinical Summarization #Fact-Level Evaluation #Evidence Grounding #LLM Hallucination
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments