ZICQ
中 Log in / Sign up
Newsroom LLMs & Foundation Models #System-1 Decision Models #LLM Agents #Model Evaluation #Performance Comparison #Robustness

System-1 Decision Models Evaluation: Comparing Laya and Jev in LLM Agent Tasks

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 12 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:This research presents a comprehensive evaluation of two System-1 decision models, the open-weight Laya and the hosted Jev, in LLM agent tasks. The study is based on 11 decision points from 18 public sources, encompassing 7,283 base cases and 6,640 robustness variants. The results show that Jev significantly outperforms Laya on 9 out of 11 decision points, with accuracy improvements ranging from 10.8% to 46.0%. However, neither model surpasses chance on zero-shot model routing, and they tie on R


Background and Motivation

In LLM agent applications, System-1 decision models are widely used for rapid decision-making due to their efficient single forward pass characteristics, such as selecting which model to call, which tool to use, evaluating the relevance of retrieved text, and detecting injection attacks in inputs. However, there is a lack of systematic evaluation and comparison of the performance of different System-1 decision models. This study aims to fill this gap by providing an in-depth comparison of two representative models, the open-weight Laya and the hosted Jev, revealing their strengths and weaknesses in practical applications.

Methodology

The study is based on 11 decision points from 18 public sources, encompassing 7,283 base cases and 6,640 robustness variants. The evaluation employs the following methods:

  • Byte-level input consistency: Ensuring input data is identical across different models.
  • Paired testing: Conducting paired tests for each case to eliminate random errors.
  • Cross-hardware and cross-day reproducibility checks: Verifying the consistency of model performance across different hardware and different times.

Key Findings

  1. Jev's Significant Advantage: Jev significantly outperforms Laya on 9 out of 11 decision points, with accuracy improvements ranging from 10.8% to 46.0%.
  2. Limitations of Zero-shot Model Routing: Neither model surpasses chance on zero-shot model routing.
  3. Tied Performance on RAG Relevance Gating: Both models perform similarly on RAG relevance gating.
  4. Laya's Stability Issues: Laya changes 30% of its answers when the option order is reversed and degrades sharply with many or similar candidates (31% at 50 nearest-neighbor tools, compared to 98% for Jev on items with a unique correct tool).

Errors and Confounds in Original Analysis

The study also identifies several errors and confounds in the original analysis:

  • Omitted Pre-screen Cost: The reported savings rate was 23.9%, but the actual rate was only 4.3%.
  • Misreporting Gate Accuracy as End-to-end Quality: The reported accuracy was 58%, but the actual accuracy was 98%.
  • In-sample Threshold Issues: The target was 5%, but there were up to 17% out-of-sample misses.
  • 'Channel Effect' on Injection False Positives: The effect disappeared with channel-native content.

Conclusion and Recommendations

This study provides new insights into the application of System-1 decision models in LLM agents, emphasizing the importance of model selection and evaluation. Developers should choose the appropriate model based on the specific application scenario and pay attention to potential biases and confounding factors in the evaluation process. Future research can further explore the application of models in different fields and how to improve model architecture and training methods to enhance their stability and robustness.


Source: ArXiv AI (cs.AI) (2026-10-06)

— END —

Tags: #System-1 Decision Models #LLM Agents #Model Evaluation #Performance Comparison #Robustness

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…