ArXiv Releases General Decision Models Benchmark: Insights from JEVal and InnerJev Models

Summary:ArXiv has released a study on general decision models, introducing JEVal, a bilingual benchmark comprising 11,257 instances from 36 datasets across 10 domains, and evaluating 25 model configurations. The results show that while general decision models excel in evidence-based decisions, they struggle with tasks requiring specialist knowledge or faithful uncertainty estimation. In dynamic systems with long-horizon, multi-step interactions, the advantages of fast local decision-making are offset by

---

Background and Motivation

In recent years, General Decision Models have emerged as efficient alternatives to LLMs for structured judgment and selection tasks. However, their performance in complex tasks and reliability in multi-step interactions remain under-explored. To address this, the research team introduced the JEVal benchmark and evaluated a variety of model configurations.

Key Research Components

1. JEVal Benchmark:

   • Comprises 11,257 instances from 36 datasets across 10 domains.
   • Evaluates 25 model configurations, including general decision models and generative LLMs.

2. Research Findings:

   • Evidence-Based Decisions: General decision models perform well in evidence-based decisions but struggle with tasks requiring specialist knowledge or faithful uncertainty estimation.
   • Dynamic Systems: In dynamic systems with long-horizon, multi-step interactions, the advantages of fast local decision-making are offset by reliability issues at the system level.
   • Large-Scale Social Simulations: Decision models approach strong generative LLMs in individual response prediction but lag in user profiling and exhibit larger aggregate estimation errors.

3. InnerJev Models:

   • Proposes InnerJev-4B and InnerJev-27B, which internalize an open-weight LLM's reasoning into a single-pass first-token decision through Reasoning-to-Readout Self-Distillation.
   • InnerJev-27B performs on par with Jev on JEVal while significantly reducing response time to about 0.1 seconds for a typical query.

Technical Highlights

• JEVal Benchmark: Provides a comprehensive evaluation framework covering multiple application scenarios and model configurations.
• InnerJev Models: Achieves efficient inference through self-distillation, significantly improving decision speed.
• Multi-Dimensional Evaluation: Assesses models from multiple perspectives, including evidence-based decisions, dynamic systems, and social simulations, offering a thorough performance analysis.

Industry Impact and Developer Recommendations

• Industry Impact:

  • General decision models hold potential in specific domains but require optimization in combination with specialist knowledge and uncertainty estimation.
  • Reliability issues in dynamic systems need further research to enhance model performance in real-world applications.

• Developer Recommendations:

  • When using general decision models, choose appropriate model configurations based on task requirements.
  • Developers can draw inspiration from the design of InnerJev models to explore methods for internalizing LLM reasoning into efficient decision processes.

Conclusion

This study provides new insights into the application of general decision models and proposes the InnerJev models, offering a new path for optimizing AI decision systems.

---

Source: ArXiv NLP/LLM (cs.CL) (2026-10-06)

— END —

Tags: #Decision Models Follow #LLMs & Foundation Models Follow #Benchmark Follow #Inference Optimization Follow #ArXiv Follow