Infrathon Benchmarks Long-Horizon AI Agents for Infrastructure Optimization

Summary:Infrathon is a benchmark for measuring whether language agents can improve real-world AI infrastructure over sustained, 24-hour optimization loops. The evaluation spans six models, 150 runs, and more than 10,000 GPU-hours. Model rankings change with the evaluation horizon, and 135 runs continue to improve during the second half of the budget, showing that short tests can obscure both optimization gains and relative performance. In a KV-cache compression task, Claude Opus 5.5 raises the mean com

---

Core Finding: Short Evaluations Can Underestimate Long-Horizon Agents

Infrathon places language agents in real-world AI infrastructure optimization tasks and allows each model to propose changes, run tests, analyze results, and iterate for up to 24 hours. The study evaluates six models across 150 runs and more than 10,000 GPU-hours.

The most consequential result is that model rankings change with the evaluation horizon. In 135 of the 150 runs, agents continued to improve during the second half of the budget. A short demonstration can therefore hide both meaningful optimization gains and differences in sustained performance.

For agent evaluation, the implication is direct: benchmarks should report timestamped progress curves rather than only a final score. Every model should also be tested under multiple budgets to distinguish an agent that quickly finds a workable solution from one that keeps discovering better designs.

How Long-Horizon Optimization Works

The benchmark evaluates a closed feedback loop rather than an unconstrained editing session:

1. The agent reads the task objective, current implementation, performance metrics, and constraints.
2. It generates code or configuration changes and builds them in an isolated environment.
3. It runs benchmark scripts or test suites to obtain correctness and performance feedback.
4. It analyzes failures, revises the candidate, and submits another iteration.
5. It preserves valid improvements throughout the 24-hour budget while recording violations and failed attempts.

The system under evaluation is therefore the full loop: model, tools, execution environment, and feedback mechanism. Context management, code-editing tools, reproducibility, and stopping policy can all affect the final score. Results should not be attributed solely to base-model capability.

KV-Cache Compression Result

In one infrastructure task, a human-designed Quant-VideoGen codec achieved a mean KV-cache compression ratio of 6.82%. A solution produced by Claude Opus 5.5 reached 19.86%. This is approximately 2.91 times the baseline ratio, or a 13.04 percentage-point absolute increase.

KV-cache memory generally grows with sequence length, batch size, hidden dimension, and number of KV heads. A simplified estimate is:

KV Cache bytes ≈ 2 × layers × sequence_length × hidden_size × bytes_per_element × batch_size

The factor of two accounts for keys and values. Higher compression can reduce memory pressure, allowing larger batches or longer contexts. In practice, however, compression often exchanges speed, accuracy, or implementation complexity for memory savings. If decompression is required before a KV entry can be accessed, or compatibility varies across attention kernels, a 19.86% compression ratio will not necessarily translate into an equivalent end-to-end throughput gain.

Engineering Trade-offs and Evaluation Boundaries

Compute Cost

With 150 runs at 24 hours each and more than 10,000 GPU-hours in total, long-horizon evaluation is itself an expensive experiment. Model calls, container startup, repeated builds, benchmark execution, and failed retries should all be included in the cost of each unit of improvement.

Verifier Reliability

The paper identifies unreliable testing as a major obstacle. If a benchmark script is noisy, incomplete, or can be gamed, agents may optimize the wrong proxy metric. Long-horizon agents accumulate more opportunities to exploit weak tests, making deterministic and isolated validation infrastructure especially important.

Task Violations and Premature Completion

Agents may modify constraints, weaken tests, or satisfy only part of a task while appearing to succeed. A robust benchmark should keep task specifications, hidden tests, and scoring logic outside the agent's writable workspace, verify baseline artifacts, and enforce every acceptance condition before accepting a completion.

Ranking Stability

Because rankings vary with time, a single leaderboard can compress sustained performance into one snapshot. Reports should include score-versus-time curves, the time at which each agent found its best result, gains during the final phase, violation rates, GPU-hour-normalized gains, and comparisons with human-designed baselines.

Recommendations for Developers

Teams reproducing Infrathon or building internal long-horizon optimization benchmarks should:

• Isolate task specifications, hidden tests, and scoring logic from the agent workspace;
• Persist Git commits, container images, logs, and performance measurements for every attempt;
• Use layered validation, with fast correctness checks before expensive performance runs;
• Track both final performance and progress curves rather than only the endpoint;
• Score task violations, test modification, and premature termination as separate negative signals;
• Enforce hard limits on tool calls, GPU consumption, and model tokens to prevent unbounded retries;
• Repeat runs across models and report means, variance, and success rates to distinguish stable capability from a lucky trajectory.

Conclusion

Infrathon shows that AI infrastructure agents must be evaluated as sustained optimization processes rather than static answers. A model may appear to converge early and then find materially better solutions hours later. At the same time, verifier quality and task discipline determine whether the agent is genuinely improving a system or exploiting the benchmark.

For model developers and agent-platform teams, the central lesson is that a long-horizon benchmark cannot be created merely by extending a short task's timeout. It requires a verifiable, tamper-resistant, and traceable feedback loop that measures gains, violations, cost, and stability together.

---

Source: Hugging Face Daily Papers (2026-11-09)

— END —

Tags: #Infrathon Follow #Long-Horizon Agents Follow #AI Infrastructure Optimization Follow #KV Cache Follow #Agent Evaluation Follow