Agent Evolution Is Not Monotonic: EvoAudit Exposes the Engineering Limits of Recursive Improvement

Summary:This study evaluates four methods for evolving agent harnesses across general and coding tasks, focusing on whether benchmark gains translate into reliable, efficient, and transferable improvements. All 16 evolved agents solved previously uncovered tasks, but they also lost tasks they had previously handled. Reliability gains often increased deployment cost, and the highest-scoring agent did not produce the largest average improvement across the four evaluation pools. The paper argues for preser

---

Core Finding: Recursive Improvement Can Forget Capabilities

This study examines agent harness evolution, a form of recursive self-improvement (RSI) that modifies an agent's prompts, tool orchestration, execution flow, or acceptance mechanisms without updating the underlying model weights. It evaluates four evolution methods across two harness types, general tasks, and coding tasks. Repeated evaluations, controlled agent expansion, component interventions, and admission replays are used to test whether apparent gains are reliable, efficient, and transferable.

The results challenge the assumption that agent evolution produces monotonic capability gains. All 16 evolved agents solved tasks that had previously been uncovered, but they also lost tasks they had previously completed. Harness evolution therefore redistributes capabilities rather than simply accumulating them. In one general-task setting, reliability increased by 5.2 percentage points while ten previously covered tasks were lost. A higher reliability score cannot, by itself, be interpreted as an overall capability improvement.

Why the Top Benchmark Score Is Not the Largest Real Improvement

The highest-scoring agent did not produce the largest mean improvement across the four evaluation pools. A conventional leaderboard rewards final pass rates but tends to hide three costs:

1. Capability forgetting: Coverage of new tasks can increase while previously solved tasks regress.
2. Reliability-cost coupling: Most reliability improvements also increase deployment cost.
3. Non-additive component effects: Specific changes generally outperform generic reminders, but combining them can remove or reverse individual benefits.

Evaluation should therefore track task-level retention, regression failures, deployment cost, and transfer across harnesses and task domains. Stricter admission gates are not automatically beneficial either: they may discard complementary candidates and produce a higher aggregate score without a larger mean improvement.

How EvoAudit Works

EvoAudit converts the study's diagnostics into a select–analyze–mutate–gate loop:

• Select: Choose representative tasks and historical traces, separating new capability coverage from regressions on previously covered tasks.
• Analyze: Attribute failures to prompts, tool calls, context organization, recovery behavior, or acceptance logic instead of editing only the visible instruction.
• Mutate: Generate harness changes aimed at concrete failure modes. Mutations must control component interactions so that individually useful changes do not degrade after composition.
• Gate: Accept or reject a candidate using task-level regression, reliability, deployment cost, and explicit applicability conditions rather than aggregate benchmark score alone.

From an implementation perspective, EvoAudit resembles automated software evolution with regression testing rather than unconstrained prompt search. Each candidate harness should be evaluated repeatedly and checked for newly covered tasks, retained tasks, token consumption, tool calls, retries, latency, and human intervention. The admission stage should retain candidate evidence so that overly strict gating does not eliminate useful alternatives.

Transfer Is Selective

Transfer results differ between general and coding tasks. For the general-task target, coverage usually falls across both harness types and evolution paradigms. For the coding target, coverage can increase with an outer-layer-only harness. Harness-level improvements are therefore not automatically transferable capabilities: task type, framework boundaries, and the location of the modification all matter.

This limits direct deployment of the findings. An optimization discovered in one tool environment should not be assumed to work in another. Coding agents, for example, interact with a different feedback loop involving test generation, compilation, patch submission, and verification scripts than general tool-calling agents.

Deployment Recommendations

Teams building continuously evolving agent harnesses should adopt the following practices:

• Maintain task-level pass matrices for every candidate instead of storing only aggregate scores.
• Report new-task coverage and existing-task retention as separate metrics, with a dedicated regression suite.
• Repeat evaluations to distinguish genuine improvements from sampling noise.
• Track token usage, tool calls, retries, end-to-end latency, and human intervention alongside accuracy.
• Run single-component ablations before testing combinations to expose interaction effects.
• Preserve rejected candidates and their task-level evidence during admission.
• Validate general tasks, coding tasks, different harnesses, and different tool environments separately.

The central contribution is not a claim that RSI always improves agents. It is a more rigorous engineering model for agent evolution: treat harness changes like software releases and continuously test new capability, capability retention, cost budgets, and transfer boundaries. Only when these dimensions improve together can recursive self-improvement move from benchmark gains to deployable system quality.

---

Source: Hugging Face Daily Papers (2026-11-05)

— END —

Tags: #Recursive Self-Improvement Follow #Agent Harness Follow #Agent Evaluation Follow #EvoAudit Follow #Capability Regression Follow