DramaGrow: A Regime-Aware Benchmark for Auditing LLM Judges of Multi-Episode Drama
Paper · https://arxiv.org/abs/2611.09938
arXiv:2611.09938v1 Announce Type: new Abstract: LLMs can now draft complete multi-episode dramas, and LLM judges are increasingly used to evaluate them. Assessing the reliability of these judges is difficult because drama quality encompasses specification-backed errors, reproducible preferences, and legitimate disagreement among human raters. Most existing evaluations reduce these cases to a single score from one judge call and leave tie calibration, call-to-call variation, and the vote margin underlying the human reference unreported. We introduce DramaGrow, a regime-aware benchmark and audit protocol for AI-generated Chinese five-episode drama. The benchmark assigns verifiable constraint violations, four-of-five preferences, and within-panel disagreements to separate denominators and evidence rules. It contains 800 episodes, 120 byte-preserving interventions, and 1,155 blinded five-rater records, including a frozen extension with 114 supermajority pairs and 21 disagreement pairs. On the frozen sets of 114 preference pairs and 35 constraint interventions, the repeated absolute-score rubrics yield overlapping performance, with mean accuracies of 53.8-55.8% and three-call agreement of 29.8-32.5%, and their accuracy falls from 71.6-71.9% to 34.8-37.1% as the tie band widens. DirectPair-GLM reaches 85.1% preference accuracy, 89.5% constraint F1, and 81.9% Detect+Locate on the same items, and exact subpanel enumeration identifies four-of-five as the smallest stable human vote margin. Evaluation protocol, tie calibration, judge-call variation, and panel margin therefore affect performance as much as rubric design. We will release the specification packages, scripts, interventions, anonymized human votes and rationales, frozen judge runs, and analysis code.
First seen by Oriel on .
AI summary
DramaGrow is a regime-aware benchmark for auditing LLM judges of AI-generated Chinese five-episode drama, showing evaluation protocol and calibration influence performance as much as rubric design.
- The benchmark includes 800 episodes, 120 byte-preserving interventions, and 1,155 blinded five-rater records, with frozen preference, disagreement, and constraint sets.
- Repeated absolute-score rubrics achieved 53.8-55.8% mean accuracy and only 29.8-32.5% three-call agreement on frozen preference and constraint sets.
- As the tie band widened, absolute-score rubric accuracy dropped from 71.6-71.9% to 34.8-37.1%.
- DirectPair-GLM achieved 85.1% preference accuracy, 89.5% constraint F1, and 81.9% Detect+Locate, while subpanel analysis found four-of-five the smallest stable human vote margin.
- The authors plan to release specifications, scripts, interventions, anonymized votes and rationales, frozen judge runs, and analysis code.
#llm judges #benchmarking #drama generation #human evaluation
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2611.09938v1 | https://arxiv.org/abs/2611.09938 Read discussion |