RealtimeBench and TORSI: A New Framework for Real-Time Agent Evaluation and Cross-Timescale Learning
Summary:This paper introduces RealtimeBench, a benchmark for evaluating agents in real-time environments with 180 tasks across six game types, designed to assess agents while games continue running during inference. The study finds that GPT-6.1-sol outperforms the faster Jev when games are paused but falls behind when they run continuously. To address this, the authors propose Two-Timescale Offline Recursive Self-Improvement (TORSI), a method where a slow teacher agent studies failures and writes reusab
A New Framework for Real-Time Agent Evaluation and Cross-Timescale Learning
1. RealtimeBench: A Benchmark for Real-Time Agent Evaluation
RealtimeBench is a novel benchmark designed to evaluate agents in real-time environments. It includes 180 tasks across six game types, simulating scenarios where the environment continues to change during inference.
2. TORSI: A New Method for Cross-Timescale Learning
To address the challenges of real-time environments, the authors propose a new method called Two-Timescale Offline Recursive Self-Improvement (TORSI). This method consists of two main phases:
- Slow Teacher Agent Phase: The teacher agent analyzes failures and tests corrections in paused games.
- Fast Student Agent Phase: The teacher agent writes reusable instructions for the student agent, which executes tasks without teacher intervention during gameplay.
3. Experimental Results and Analysis
Experiments show that GPT-6.1-sol outperforms the faster Jev when games are paused but falls behind when they run continuously. The TORSI method significantly improves Jev's average test score from 13.51 to 24.19 on a 0–100 progress scale across all six game types. Additionally, the instructions also enhance the performance of three other models without retuning, demonstrating its broad applicability.
4. The Potential of Real-Time Intelligence
These findings pave the way for real-time intelligence, enabling powerful LLM agents to remain effective in a continuously changing world by organizing deliberation, learning, and action across timescales. The TORSI method provides a new path for AI agents to efficiently deploy in complex tasks.
5. Engineering Trade-offs and Challenges
While the TORSI method shows promise in improving agent performance, its implementation also presents some challenges. For example, the training and inference processes of the slow teacher agent require high computational resources, while the instruction writing and execution of the fast student agent require efficient algorithm design. Furthermore, maintaining the stability and robustness of the agent in dynamic environments is another area that requires further research.
Developer Recommendations
- Resource Management: When deploying the TORSI method, it is recommended to allocate computational resources according to task requirements to balance performance and cost.
- Instruction Optimization: The instructions written by the teacher agent should be as concise and clear as possible for the fast student agent to execute efficiently.
- Continuous Monitoring and Adjustment: It is recommended to continuously monitor the agent's performance and adjust instructions and strategies according to environmental changes.
— END —Source: ArXiv AI (cs.AI) (2026-10-17)
Tags: #RealtimeBench #TORSI #Agent Evaluation #Cross-Timescale Learning #Real-Time Intelligence
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments