DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems
Paper · https://arxiv.org/abs/2610.28522
Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make correct decisions across controlled variations or assign reliable confidence to their predictions. We introduce DecisionPivot, a diagnostic benchmark comprising 48 tasks and 40,033 items across seven decision roles. Its core track contains 7,800 items from 13 generated task families, organized into contrast groups with computed answers verified by independently implemented solvers. The complementary reference track contains 32,233 items from 35 public test sets, preserving their original evaluation metrics. We evaluate 40 decision models and 14 general models using documented, model-specific protocols, assessing item and contrast-group accuracy, probability ordering, calibration, and empirical selective coverage. Our results reveal substantial differences obscured by aggregate scores: two 27B models separated by only 2.6 points on the reference track achieve contrast-group accuracies of 70.1% and 44.2%. Across 2,400 binary contrast pairs, Perplexity Decider correctly ranks the positive member above the negative member in 92.3% of pairs but answers both members correctly in only 74.1%, exposing a gap between probability ordering and decision correctness. Models with similar item accuracy also differ in the reliability of their high-confidence predictions. On a separate 1,296-item subset, GLM-5.3 achieves 93.5\% contrast-group accuracy at a higher reasoning setting, compared with 81.5% at its lowest effort. By linking errors to pivotal details and difficulty levels, DecisionPivot provides fine-grained diagnostics of decision failures that aggregate evaluation alone cannot reveal.
First seen by Oriel on .
AI summary
DecisionPivot evaluates whether models in LLM systems make reliable decisions across controlled variations, finding that aggregate accuracy can hide substantial differences in correctness, probability ordering, and confidence.
- The benchmark spans 48 tasks and 40,033 items across seven decision roles, with a 7,800-item core track and a 32,233-item reference track.
- Its core track uses 13 generated task families, contrast groups, and answers verified by independently implemented solvers; the reference track preserves metrics from 35 public test sets.
- Evaluation of 40 decision models and 14 general models measures item and contrast-group accuracy, probability ordering, calibration, and empirical selective coverage.
- Two 27B models only 2.6 points apart on the reference track score 70.1% and 44.2% on contrast-group accuracy.
- Perplexity Decider ranks positive examples above negative ones in 92.3% of 2,400 pairs but answers both correctly in 74.1%; GLM-5.3 contrast-group accuracy also rises from 81.5% to 93.5% at a higher reasoning setting.
#llm evaluation #decision models #benchmarking #calibration
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hugging Face Papers: 74, latest observation ; +61 over observed period since
Comments
Comment count reported by the source.
- Hugging Face Papers: 3, latest observation ; +2 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Papers | 2610.28522 | https://arxiv.org/abs/2610.28522 Read discussion |