arXiv Releases CTI-REALM Study: Open-Weight Models Demonstrate Cost Advantage in Security AI Agents
Summary:arXiv has released a study on CTI-REALM, examining the performance and cost trade-offs between open-weight and closed models in security AI agents. The research demonstrates that open-weight models can match the strongest closed models at less than a tenth of their cost when given sufficient turns. The study introduces stage-level metrics to evaluate agent submissions by executing detection queries in a telemetry store and suggests routing workflow stages to the best-performing model for each ta
Background and Objectives
In the field of security, language model agents are widely used for threat intelligence processing and detection tasks. However, there is a significant cost and performance gap between open-weight and closed models. This study aims to evaluate the performance of these two types of models in the CTI-REALM benchmark and explore their cost-effectiveness.
Technical Mechanism Analysis
CTI-REALM is a benchmark that transforms threat intelligence reports into actionable detection results, with agents completing this task by querying live telemetry data. The research team evaluated 26 models, including 19 open-weight and 7 closed models, using a unified evaluation framework, scorer, and panel of LLM judges.
The core finding is that open-weight models can match the strongest closed models at less than a tenth of their cost when given sufficient turns. The traditional message limit causes tasks to score zero upon reaching the limit, restricting the model's performance. With the limit relaxed, previously truncated models recover, and open-weight models show higher rewards.
Engineering Trade-offs and Performance
The study introduces stage-level metrics, evaluating model submissions by executing detection queries in telemetry data and comparing them against the answer key. This approach reveals the strengths of different models at various stages, suggesting that selecting the best model for each task stage is more effective than relying on a single overall score. Additionally, the study reports the number of tokens each model generates, finding that the share of output spent on thinking does not correlate with the reward.
Developer Deployment Recommendations
For developers in the security field, this study offers the following recommendations:
- Prioritize Cost-Effectiveness: In budget-constrained environments, open-weight models are a more economical choice, especially for complex tasks requiring extended runtime.
- Stage-Based Model Selection: Choose the best model for each task stage rather than relying on a single model for the entire workflow.
- Relax Message Limits: When evaluating models, consider relaxing message limits to more comprehensively assess model performance.
Conclusion
This study demonstrates the significant potential of open-weight models in the security AI agent domain, particularly in terms of cost-effectiveness. By introducing stage-level metrics and relaxing message limits, the research provides new perspectives and methods for future model selection and evaluation.
— END —Source: ArXiv AI (cs.AI) (2026-10-15)
Tags: #CTI-REALM #Open-Weight Models #Security AI Agents #Cost-Effectiveness #LLMs & Foundation Models
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments