LHTB 2.0 Released: Revolutionizing Long-Horizon Agent Evaluation with Sustained Task Performance Focus

Summary:Long-Horizon-Terminal-Bench 2.0 (LHTB 2.0) is an enhanced version of the existing long-horizon agent evaluation benchmark, preserving 46 original tasks while addressing task and infrastructure defects and adding 31 new tasks, totaling 77 tasks across 13 domains and eight task types. LHTB 2.0 introduces parameterized reference implementations, behavioral evidence generation, and dense progress ladders to more accurately measure sustained agent performance in long-horizon tasks, supporting evaluat

---

Core Breakthroughs

Long-Horizon-Terminal-Bench 2.0 (LHTB 2.0) is a significant upgrade to the existing long-horizon agent evaluation benchmark, featuring the following key aspects:

1. Task Expansion and Correction: Retaining 46 original tasks, LHTB 2.0 corrects defects in tasks and evaluation infrastructure and adds 31 new tasks, totaling 77 tasks across 13 domains and eight task types.

2. Enhanced Evaluation Mechanism: Introduces parameterized reference implementations and dense progress ladders to more accurately measure sustained agent performance in long-horizon tasks.

3. Extended Evaluation Time: Extends the evaluation horizon from 90 minutes to three hours and records checkpoint anchors every 30 minutes to directly measure sustained progress within a run.

4. Behavioral Evidence and Reward Mechanisms: Generates behavioral evidence and uses dense progress ladders to reward partial progress while preventing memorization of public outputs.

Technical Mechanism Analysis

The core of LHTB 2.0 lies in its task generation and evaluation pipeline:

• Task Generation: Starting from a working scientific or engineering artifact, LHTB 2.0 uses a parameterized reference implementation to generate multiple scenarios, selects six as behavioral evidence, and retains three for evaluation.

• Evaluation Method: Employs dense progress ladders to reward partial progress rather than relying solely on the final state binary check. This approach captures agent performance in tasks more comprehensively.

• Evaluation Infrastructure: Records checkpoint anchors every 30 minutes to assess sustained progress in long-horizon tasks.

Engineering Trade-offs and Performance

LHTB 2.0 optimizes in several areas but also faces trade-offs:

• Evaluation Time and Resource Consumption: Extending the evaluation time to three hours increases the comprehensiveness of the assessment but also raises the demand for computational resources.

• Task Complexity: The addition of new task types and domains increases the complexity of the evaluation, allowing for a more accurate assessment of agent performance in diverse tasks but also posing higher requirements for the agent's generalization capabilities.

Experimental results show that GPT-6-astra leads in LHTB 2.0, demonstrating its superiority in complex tasks.

Developer Implementation and Deployment Recommendations

For developers, LHTB 2.0 provides a more comprehensive and rigorous testing platform for evaluating agent performance in long-horizon tasks. It is recommended that developers:

• Fully utilize LHTB 2.0's evaluation mechanisms to comprehensively test agent performance across diverse tasks.

• Focus on sustained progress in long-horizon tasks, rather than relying solely on the binary check of the final result.

• Combine LHTB 2.0's behavioral evidence and reward mechanisms to optimize the task execution strategy of the agent.

Conclusion

The release of LHTB 2.0 marks a significant advancement in long-horizon agent evaluation benchmarks, providing a more comprehensive and rigorous testing platform for studying reliable agent execution over extended periods.

---

Source: ArXiv AI (cs.AI) (2026-10-15)

— END —

Tags: #Long-Horizon Agent Follow #Evaluation Benchmark Follow #GPT-6 Follow #Artificial Intelligence Follow #Reinforcement Learning Follow