SO-Arena Benchmarks 21 Fast-Decision Models: Accuracy Does Not Guarantee Reliability
Summary:SO-Arena evaluates fast, non-long-chain-of-thought System One models across 11 domains and five capability families, separating capability breadth from decision reliability. Its protocol combines evidence-grounded labels, domain-specific difficulty criteria, sandboxed interaction, perturbation-based stability tests, and missing-answer recognition. Results across 21 models show that scaling and additional reasoning yield domain-dependent gains. More importantly, the highest-performing System One
What the Benchmark Measures
SO-Arena focuses on System One models, which are designed to make fast, general-purpose decisions without generating long reasoning chains. Conventional evaluations often collapse performance across domains and difficulty levels into a single average score. That score cannot reveal whether a model has broad capability coverage, remains consistent when answer options change, or recognizes when no valid answer exists.
The benchmark spans 11 domains, five capability families, and 21 models. Its central contribution is to separate capability breadth from decision reliability rather than collapsing both into one ranking.
Measuring the Capability Spectrum
The first component is evidence-grounded evaluation. Reference answers are drawn from reliable sources and manually verified against the information visible to the model. This reduces label noise and avoids penalizing a model when the necessary evidence was never available in its input context.
The second component introduces domain-specific difficulty criteria. Instead of assigning only a binary correct or incorrect label, SO-Arena can distinguish performance across tiers such as basic recognition, compositional reasoning, and difficult decision-making. The result resembles a capability matrix: teams can identify not only where a model fails, but also whether the failure originates in foundational competence or in more demanding combinations.
Third, the benchmark supports realistic interaction. Models iteratively execute actions in sandboxes, and the environment advances based on execution results rather than static text alone. This is more representative of tool-using agents because it measures whether a model can turn a decision into an action and adjust after environmental feedback.
Some reasoning-intensive tasks also provide System One–System Two comparisons. These comparisons show how additional reasoning compute changes outcomes, although they do not by themselves establish that the observed gain comes from a particular internal cognitive mechanism.
Two Complementary Reliability Measures
SO-Arena deliberately does not treat accuracy as a substitute for reliability. Instead, it introduces two complementary metrics.
Decision Stability
The evaluation changes the option set while keeping the underlying task controlled, for example by adding or removing distractors. It then measures whether the model preserves its decision. A model that changes its answer under semantically equivalent perturbations may be relying on option position, surface wording, or dataset-specific patterns rather than a stable task-relevant representation.
From an engineering perspective, this should be implemented as a perturbation test: retain the original instance and evidence, modify only the candidate set, and report consistency, flip rate, and post-perturbation accuracy. Raw accuracy alone can reward lucky initial selections without revealing whether a model can survive the option changes encountered in production.
Missing-Answer Recognition
The second metric asks whether a model can abstain or identify insufficient information when the candidate set contains no valid answer. This is closer to calibration than generic refusal behavior. The desired behavior is not constant caution, but abstaining when no answer is supportable while remaining decisive when sufficient evidence exists.
A model that guesses across no-answer samples—or refuses even when a valid option exists—may still obtain a respectable average accuracy score, but it is poorly suited to high-cost decision workflows.
Results Across 21 Models
The most important finding is that the benefits of scaling and additional reasoning vary substantially by domain. More inference compute does not improve every task proportionally, nor does it produce a stable ordering between fast and long-reasoning systems. Domain characteristics, dependence on evidence, interaction feedback, and candidate-space complexity all change the value of a reasoning budget.
More significantly, the highest-performing System One model ranks only seventh in reliability. Aggregate accuracy and dependable decision-making are not interchangeable. A model may achieve a strong overall score by recognizing answer patterns or exploiting dataset-specific cues, then fail under controlled option perturbations or force an answer when none is valid.
Model selection should therefore report more than one mean score. A useful evaluation includes domain-stratified accuracy, calibration on no-answer cases, consistency under perturbation, and interactive task completion, weighted according to the cost of each failure mode.
Engineering and Evaluation Trade-offs
SO-Arena is closer to real systems than static question answering, but it also costs more. Sandbox execution requires environment state management, model-action execution, and reproducible resets, making it slower than one-shot generation. Evidence verification and manual label review raise dataset-preparation costs. The benefit is that failures can be traced through concrete environment interactions instead of appearing only as an aggregate score.
The metrics also have boundaries. Decision stability depends on whether perturbations preserve task semantics and the equivalence of valid answers; arbitrary option changes can accidentally create a different or unfair problem. Missing-answer recognition is sensitive to refusal wording, prompt templates, and answer parsers. A keyword-only refusal detector can create systematic bias across models.
Testing 21 models is enough to expose broad trends, but not enough to prove that every conclusion generalizes to all model families. If model size, training data, reasoning budget, system prompts, and tool permissions are not controlled, performance differences cannot be attributed solely to System One architecture. Cross-paper comparisons must align sampling parameters, context limits, tool access, and interaction policy.
Practical Adoption
Teams can use SO-Arena as a reference framework for model selection and release gating rather than a drop-in replacement for internal evaluation. A practical three-stage process is:
- Build a capability profile. Evaluate models by domain and difficulty to verify that business-critical abilities are covered rather than hidden by an average score.
- Test stability. Apply real-world perturbations such as paraphrases, distractor insertion, and option-order changes, then track answer flips and confidence shifts.
- Calibrate abstention. Construct explicit no-solution, insufficient-information, and conflicting-evidence cases to distinguish justified abstention from excessive caution and dangerous guessing.
For agent workloads, sandbox failure traces should also be monitored. A model may remain stable on single-turn questions yet fail to revise actions based on tool feedback. Conversely, long-reasoning models may be more reliable on complex tasks while adding latency and token cost. System One models are best suited to paths with explicit latency budgets, recoverable errors, and actions that can be validated through environmental feedback.
Conclusion
The key lesson from SO-Arena is that a fast model's value cannot be defined by average accuracy alone. Fast decision-making is operationally useful only when capability coverage is adequate, decisions remain stable under perturbation, and the model recognizes its knowledge boundary when no valid answer exists. By separating “can it answer” from “can it repeatedly make trustworthy decisions,” the benchmark provides a more rigorous framework for evaluating low-latency models, agent controllers, and hybrid reasoning systems.
— END —Source: Hugging Face Daily Papers (2026-11-11)
Tags: #SO-Arena #System One Models #Decision Reliability #Model Evaluation #Fast Inference
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments