SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models
Paper · https://arxiv.org/abs/2611.15777
System One models promise fast, general-purpose decision-making without lengthy token generation. However, understanding their capability spectrum and decision reliability requires more than aggregate performance on current benchmarks. Existing evaluations often provide limited insight into the breadth of model capabilities and the reliability of their decisions. We introduce SO-Arena, a benchmark characterizing the capability spectrum and decision reliability of System One models across 11 domains and five capability families. For the capability spectrum, SO-Arena offers evidence-grounded evaluation through reliable sources and manual verification of reference labels against model-visible inputs; fine-grained capability assessment through domain-specific difficulty criteria; and realistic interactive evaluation, where models iteratively execute actions in sandboxes. Reasoning-intensive tasks further enable system one-two comparison. Beyond accuracy, SO-Arena assesses decision reliability through two complementary measures: decision stability under controlled option-set changes and missing-answer recognition when no valid option is available. Evaluations across 21 models reveal that the benefits of scaling and additional reasoning vary substantially across domains. More importantly, strong overall performance does not necessarily imply reliable decision-making: the best-performing System One model ranks only seventh in reliability. Together, these findings highlight the need to characterize capability spectrum and decision reliability as complementary dimensions, providing a systematic view of the strengths and limitations of System One models.
First seen by Oriel on .
AI summary
SO-Arena benchmarks System One models across capability breadth and decision reliability, finding that strong overall performance does not necessarily indicate reliable decisions.
- The benchmark covers 11 domains and five capability families using evidence-grounded labels, domain-specific difficulty criteria, and interactive sandbox tasks.
- It evaluates reliability through decision stability under controlled option-set changes and recognition that no valid answer is available.
- Reasoning-intensive tasks support comparisons between System One and System Two capabilities.
- Evaluations of 21 models show that scaling and additional reasoning provide substantially different benefits across domains.
- The best-performing System One model ranked only seventh for reliability, separating aggregate performance from dependable decision-making.
#system one #benchmarking #model reliability #ai evaluation
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hugging Face Papers: 136, latest observation ; +65 over observed period since
Comments
Comment count reported by the source.
- Hugging Face Papers: 3, latest observation ; +1 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Papers | 2611.15777 | https://arxiv.org/abs/2611.15777 Read discussion |