CogBattery Evaluates 54 Open-Weight Models on Human-Like Cognition
Summary:CogBattery adapts 13 human cognitive paradigms into text-based tasks and compares 54 open-weight language models against condition-level human data drawn from the original studies. The models often reproduce the direction of human effects while making more false alarms, and their reported confidence generally separates correct from incorrect answers less reliably than human confidence does. Across tasks, model scores share two stable dimensions. One is strongly associated with model scale, but
Matching Human Effect Directions Is Not the Same as Matching Human Behavior
CogBattery adapts 13 classic cognitive paradigms into text-based tasks and evaluates 54 open-weight language models against condition-level human data collected in the original studies. Its central question is not whether a model produces a high average score, but whether it reproduces human response patterns, error types, and confidence behavior under controlled conditions.
The n-back results illustrate the distinction. Like humans, the models responded more strongly to lures than to controls, indicating that they could detect a repeat relation in the adapted task. Yet they produced many more false alarms on ordinary non-targets than humans did. Reproducing the direction of a human effect can therefore reflect the use of broad surface cues rather than faithful reconstruction of human working memory, target maintenance, or conflict monitoring.
Confidence results expose another gap. For most models, reported confidence separated correct from incorrect answers less effectively than human confidence. Many models consistently selected the same wager, even when their classification accuracy was relatively strong. Average accuracy alone would conceal this kind of degradation, making condition-level error distributions and confidence calibration essential diagnostic signals.
Evaluation Design: From Effect Detection to Error-Structure Matching
The key design choice is to preserve experimental conditions and their corresponding human references rather than collapsing every paradigm into a single aggregate score. Each model is tested in a context aligned with the source study, after which its response distribution, error pattern, and confidence are compared with human data.
This supports three distinct levels of evidence:
- Directional reproduction: The model reacts differently across conditions in the same direction as humans.
- Mechanistic approximation: Its false alarms, misses, confidence, and cross-condition relationships resemble those of humans.
- Surface fitting: It achieves a similar result by exploiting cues unrelated to the task's intended cognitive mechanism.
For example, a stronger lure response in n-back shows that a model detects some form of repetition, but it does not by itself establish human-like state updating. If humans mainly increase false alarms under conflict, while models over-alarm on every non-target, the two systems can share a directional metric while remaining behaviorally distinct.
Cross-Task Structure: Scale Matters, but Is Not a Reliable Path to Human-Likeness
Across tasks, model scores share two dimensions, one of which is closely tied to scale. This is unsurprising: larger models generally interpret instructions more accurately, recognize stimulus relationships, and maintain task state across longer or more complicated inputs.
However, larger models are not consistently closer to humans. Scale may improve average task performance without monotonically improving human-like behavioral consistency. A smaller model can produce errors caused by poor instruction understanding, while a larger model may develop a distinctly non-human error profile, such as excessive alarms, an unusually invariant wagering strategy, or weak separation between confidence and correctness.
The study also finds no reliable predictive advantage for predefined cognitive groupings over random groupings with the same number of models. Labels such as “reasoning,” “memory,” or “language” therefore do not provide a dependable basis for predicting performance on a separate cognitive paradigm.
Engineering Implications
First, evaluation pipelines should retain condition-level metrics. Beyond aggregate accuracy, teams should record condition-specific responses, false-alarm and miss rates, confidence calibration, and sensitivity to prompt or task-structure changes. A strong subgroup average does not imply a matching error mechanism.
Second, human similarity should be tested through converging evidence rather than a single effect direction. A stronger test asks whether several mutually constraining conditions produce comparable patterns and whether errors and confidence move together as they do for humans. This is particularly useful when checking whether reinforcement learning has merely learned a test-specific shortcut or whether instruction tuning changed behavior beyond output formatting.
Third, parameter count should not be used as a proxy for cognitive mechanism. Larger models may still be worth deploying, but mechanistic claims require cross-task, cross-condition, and cross-prompt stability tests, with uncertainty intervals for the human references rather than one aggregate score that hides disagreement.
Limitations and Future Validation
Many classic cognitive paradigms were originally visual, auditory, or motor experiments. Text adaptation necessarily changes stimulus encoding, response format, and some task semantics. The findings should therefore be interpreted as comparisons between models and human references in the adapted tasks, not as universal claims about natural human cognition.
The study covers 54 open-weight models across multiple families, but training data, instruction tuning, and scale may still confound some comparisons. Public task generators, condition-level responses, calibration metrics, and statistical tests would make the results easier to reproduce and help isolate the effects of scale and cognitive grouping.
CogBattery's main contribution is to move evaluation beyond answer correctness. Human-like behavior requires matching not only the direction of effects, but also their errors, confidence structure, and dependence on experimental conditions.
— END —Source: Hugging Face Daily Papers (2026-10-30)
Tags: #CogBattery #Open-Weight Models #Cognitive Evaluation #Confidence Calibration #Human Alignment
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments