David-DL-Space Releases Paired Evaluation of System-1 Decision Models: Laya vs. Jev
Summary:David-DL-Space has released a comprehensive evaluation of System-1 decision models, comparing the open-source Laya and hosted Jev models across 11 agent decision points. The study, based on 7,283 base cases and 6,640 robustness variants, includes byte-identical inputs, paired tests, and reproducibility checks across hardware and days. Results show Jev significantly outperforms Laya on 9 out of 11 decision points, though neither model surpasses chance on zero-shot model routing, and they tie on R
Background and Motivation
As large language model (LLM) agents are increasingly applied to complex tasks, System-1 decision models have gained popularity for their efficient single-pass forward propagation mechanism, which reduces computational costs and latency. These models make many small decisions per task, such as selecting which model to call, which tool to use, assessing the relevance of retrieved text, and detecting if an input contains malicious injections.
Methodology
The David-DL-Space team conducted a systematic comparison of the open-source Laya model and the hosted Jev model. The study was based on 11 agent decision points constructed from 18 public data sources, comprising 7,283 base cases and 6,640 robustness variants. The evaluation included byte-identical inputs, paired tests, and reproducibility checks across hardware and days to ensure the reliability of the results.
Key Findings
- Performance Comparison: The Jev model significantly outperformed the Laya model on 9 out of 11 decision points, with margins ranging from 10.8 to 46.0 percentage points.
- Zero-shot Model Routing: Neither model surpassed random probability on zero-shot model routing.
- RAG Relevance Gating: Both models performed similarly on this task.
- Robustness Tests: Laya's answers changed by 30% when the option order was reversed, and its performance degraded sharply with many or similar candidates (31% accuracy at 50 nearest-neighbor tools, compared to 98% for Jev on items with a unique correct tool).
Limitations of the Study
The research team identified several analysis errors and design confounds during self-review:
- An omitted pre-screen cost led to an initial reported savings rate of 23.9%, which was actually 4.3%.
- Gate accuracy was misreported as end-to-end quality (58% vs. 98%).
- In-sample thresholds led to up to 17% misses on the held-out set.
- A 'channel effect' on injection false positives disappeared with channel-native content.
Conclusion and Impact
This study provides valuable insights into the practical application of System-1 decision models. While Jev outperformed Laya on most decision points, Laya showed advantages in certain specific scenarios. The research emphasizes the importance of considering model robustness under different conditions and calls for the consideration of these factors in model deployment. Additionally, the study demonstrates the importance of transparent evaluation through open data and code, offering new research directions for the AI community.
Recommendations for Developers
- Model Selection: When choosing a System-1 decision model, weigh the specific task requirements and scenario characteristics.
- Robustness Testing: Conduct thorough robustness testing before deploying the model, especially for scenarios involving changes in option order and candidate quantity.
- Transparent Evaluation: Encourage developers to make data and code publicly available for more transparent evaluation and comparison.
— END —Source: ArXiv AI (cs.AI) (2026-10-05)
Tags: #System-1 Decision Models #LLM Agents #Model Evaluation #Robustness Testing #AI Research
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments