arXiv Releases Long-Horizon-Terminal-Bench 2.0: Revolutionizing Sustained Agent Progress Evaluation
Summary:arXiv has released Long-Horizon-Terminal-Bench 2.0 (LHTB 2.0), a benchmark platform for evaluating sustained progress in long-horizon tasks by agents. Compared to its predecessor, LHTB 2.0 fixes task and evaluation infrastructure defects and extends 31 tasks, totaling 77 tasks across 13 domains and eight task types. It introduces dense progress ladders and records anchor checkpoints every 30 minutes, enabling precise measurement of sustained progress within a run. Experiments show that while exi
Overview
Long-Horizon-Terminal-Bench 2.0 (LHTB 2.0), released by arXiv, is a benchmark platform for evaluating sustained progress in long-horizon tasks by agents. Building on the original 46 tasks, LHTB 2.0 fixes task and evaluation infrastructure defects and extends 31 tasks, totaling 77 tasks across 13 domains and eight task types.
Technical Mechanism Analysis
The core innovations of LHTB 2.0 include:
- Task Extension and Repair: Through a systematic reconstruction-task pipeline, LHTB 2.0 not only repairs defects in existing tasks but also adds new tasks, providing a wider range of task coverage.
- Dense Progress Ladders and Checkpoint Recording: It introduces dense progress ladders and records anchor checkpoints every 30 minutes, enabling more precise measurement of sustained progress within a run.
- Process-Aware Evaluation: By rewarding partial progress and preventing memorization of public outputs, LHTB 2.0 can more fairly evaluate the process awareness of agents.
Engineering Trade-offs and Performance
In 1,145 model-task runs, GPT-6-astra leads with a dense mean reward of 0.69 and 37 tasks at R at least 0.95, followed by GPT-5.6-sol with a dense mean reward of 0.67 and 32 tasks at the same threshold.
On the 30 reconstruction tasks, binary anchor scores average 0.221, whereas dense ladder scores average 0.653. This indicates that outcome-only evaluation discards substantial intermediate progress, while LHTB 2.0's evaluation method provides a more comprehensive reflection of agent performance.
Developer Implementation and Deployment Recommendations
LHTB 2.0 offers researchers and developers a more rigorous and comprehensive testing environment to evaluate and improve agent performance in long-horizon tasks. Developers are advised to focus on the following aspects:
- Task Diversity: Utilize the diverse tasks in LHTB 2.0 to test agent performance across different domains and task types.
- Sustained Evaluation: Pay attention to the sustained progress of agents in long-duration tasks, not just the final results.
- Process Awareness: Improve the process awareness of agents through LHTB 2.0's evaluation method, enabling them to better handle complex tasks.
Conclusion
The release of LHTB 2.0 marks an important milestone in the field of agent evaluation. It not only provides a wider and more rigorous task coverage but also introduces new evaluation methods that can more comprehensively reflect agent performance.
— END —Source: ArXiv AI (cs.AI) (2026-10-14)
Tags: #arXiv #Agent Evaluation #Long-Horizon Agents #Benchmark #Process Awareness
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments