DramaGrow Audits LLM Drama Judges and Exposes Unreliable Scoring
Summary:DramaGrow introduces a regime-aware benchmark and audit protocol for LLM judges evaluating Chinese five-episode dramas. It separates verifiable constraint violations, four-of-five human preferences, and legitimate panel disagreements into distinct denominators and evidence rules. Across 800 episodes, 120 byte-preserving interventions, and 1,155 blinded five-rater records, repeated absolute-score rubrics achieve only 53.8–55.8% accuracy and 29.8–32.5% three-call agreement. DirectPair-GLM reaches
DramaGrow separates judging reliability from a single score
DramaGrow addresses a weakness in LLM-as-a-judge evaluation for Chinese five-episode dramas: one aggregate score can hide very different error regimes. The benchmark separates verifiable constraint violations, four-of-five human preferences, and legitimate disagreements within a rating panel, assigning each a distinct denominator and evidence rule.
The dataset contains 800 episodes, 120 byte-preserving interventions, and 1,155 blinded five-rater records. Its frozen extension includes 114 supermajority preference pairs and 21 disagreement pairs. The central design choice is to avoid treating objective defects, reproducible subjective preferences, and irreducible annotator disagreement as interchangeable errors.
Repeated absolute scoring is unstable
Absolute-score rubrics ask a judge to score candidates independently and then infer a winner or violation from those scores. DramaGrow shows that this design mixes judge bias, call-to-call randomness, and arbitrary tie handling.
On the frozen set of 114 preference pairs and 35 constraint interventions, repeated absolute-score rubrics achieve mean accuracies of only 53.8–55.8%. Their three-call agreement ranges from 29.8% to 32.5%. Accuracy drops from 71.6–71.9% to 34.8–37.1% as the tie band widens, indicating that calibration boundaries can materially alter reported performance.
DirectPair-GLM evaluates candidates through direct pairwise comparison and uses verifiable evidence for constraint checks. The paper reports 85.1% preference accuracy, 89.5% constraint F1, and 81.9% Detect+Locate performance. Exact enumeration of rater subpanels identifies four out of five as the smallest stable human vote margin.
The engineering trade-off is more than prompt quality
Pairwise evaluation avoids requiring a judge to maintain a stable absolute quality scale, but it introduces its own costs. A preference evaluation must inspect two candidates instead of one, while violation localization requires the judge to produce a verifiable reference. Even pairwise judging remains vulnerable to position bias, verbosity bias, genre preference, and changes in the composition of the reference panel.
The byte-preserving interventions are useful for controlled local testing, but they do not establish that a judge understands global narrative quality. Likewise, the four-of-five majority threshold is specific to the collected five-rater structure and should not be assumed to transfer to every creative domain. Production evaluation systems should therefore preserve call-level outputs, repeat calls, report agreement and confidence intervals, and run sensitivity analysis over tie thresholds.
Deployment guidance
- Maintain separate denominators for hard constraints, panel-majority preferences, and legitimate disagreements.
- Run at least three independent judge calls for open-ended outputs and report variance rather than only the aggregated score.
- Measure performance under multiple tie bands to expose threshold-driven ranking changes.
- Require traceable evidence, such as constraint identifiers, quoted spans, and violation locations.
- Prefer deterministic checks for machine-verifiable requirements and reserve LLM judges for semantic or subjective criteria.
- Test candidate-order swapping and position bias when adopting a pairwise protocol.
- Re-estimate human consensus thresholds on each target domain rather than generalizing the four-of-five result.
What the benchmark reveals
DramaGrow's significance lies less in raising a reported score from roughly 55% to 85.1% than in demonstrating that a single judge score is not a stable fact. Evaluations of generated dramas, code, reports, and other open-ended artifacts should disclose their sample regimes, tie calibration, call variance, reference-panel margins, and evidence chain. Without those details, a high score may describe the evaluation protocol rather than the evaluated output.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-11-10)
Tags: #LLM-as-a-Judge #Drama Evaluation #Preference Auditing #Human Agreement
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments