Coding Agents Solve Early but Keep Working: Termination Lag Consumes 35–60% of Steps
Summary:The paper Termination Lag measures how long coding agents continue working after they have already produced a correct repository patch. Across five models, two agent frameworks, and three SWE-bench datasets, the authors recorded and tested every distinct patch generated during each run. They found that 38–82% of successful runs solved the task with their initial source-code edit, yet Qwen3-Coder devoted a median 35–37% of its successful-run steps to post-solution activity; for Laguna-XS.2, the f
Coding Agents Solve Early but Keep Working
The paper Termination Lag isolates an overlooked operating metric for coding agents: how many steps an agent continues to execute after producing its first patch that passes the paper's correctness tests. The study covers five models, two agent frameworks, and three SWE-bench datasets. Instead of evaluating only the final submission, it records and tests every distinct repository patch produced during a run, allowing the authors to reconstruct the gap between when the task was solved and when the agent stopped.
Measuring termination lag
For an agent trajectory, the key quantities can be represented as follows:
T_first_correct: the first repository patch that passes the paper's correctness criterion;T_end: the point at which the agent framework terminates the run;T_lag = T_end - T_first_correct: the termination lag in agent steps.
The paper also converts the number of post-solution steps into a share of the complete run. Testing every distinct patch is more informative than scoring only the final commit. It identifies the step at which the task was first solved and exposes trajectories that were correct at an intermediate point but incorrect at termination. Such regressions are invisible to conventional final-submission evaluation.
A step is an agent-framework execution unit and may encompass model inference, tool calls, or one iteration of the agent loop. The abstract does not explain how step semantics were normalized across the two frameworks. The percentages are therefore better interpreted as behavioral indicators than as directly comparable token, GPU-second, or monetary costs.
Main empirical results
| Observation | Reported result | Scope | |---|---:|---| | Successful runs whose initial source-code edit was already correct | 38–82% | Successful runs across models, frameworks, and datasets | | Post-solution share for Qwen3-Coder | Median 35–37% | Successful runs on SWE-bench Verified and Multilingual | | Post-solution share for Laguna-XS.2 | Median 54–60% | Successful runs on SWE-bench Verified and Multilingual | | Continued activity that undid an earlier correct patch | 1–8% | Cases reported by the paper |
These figures measure different things. The 38–82% range is not a task success rate; it is the proportion of already-successful runs that reached a correct solution on their first source-code edit. The 35–60% medians are conditioned on successful runs and may therefore suffer from survivorship bias, since trajectories that never solved the task are excluded. The abstract also does not specify the denominator behind the 1–8% figure. The safest conclusion is that post-solution activity is not merely inefficient: in a small but consequential set of trajectories, it actively destroys an earlier correct result.
Laguna-XS.2's larger post-solution share should not automatically be attributed to its foundation model. Model behavior may be confounded by the agent framework, prompt, tool permissions, and stopping policy. Without a full factorial ablation across models and frameworks, the experiment cannot cleanly separate model-level from harness-level effects.
Why agents keep running after solving the task
The abstract reports that most additional steps are spent verifying work and exploring the repository. In a typical coding-agent loop, this corresponds to a familiar sequence: inspect the repository, generate a diff, apply it, run tests, interpret the output, and modify the repository again. Even after the tests pass, the agent generally cannot prove from its current context that no further improvement is possible.
1. The verifier supplies only partial evidence
SWE-bench tests and repository test suites can expose explicit regressions, but they may not cover hidden requirements, boundary conditions, type contracts, or maintainability constraints. When a test fails, the agent is prompted to keep editing. When it passes, the agent may still repeat checks because it cannot distinguish a complete acceptance criterion from a merely visible green build.
2. Stopping policies are often designed around failure
Many harnesses stop after no new progress, repeated failed attempts, or budget exhaustion. They do not explicitly recognize that a candidate patch has already passed acceptance. The agent can consequently explore alternative implementations, widen the refactoring scope, or rerun equivalent checks even though the task is already complete.
3. Later edits can overwrite a correct state
The context may still contain assumptions from before the first repair, and subsequent exploration may introduce unrelated changes. A previously correct file is then overwritten and the test suite fails again. The reported 1–8% regression rate shows that termination lag affects trajectory stability and final correctness, not merely operating cost.
4. Post-solution exploration is not always wasteful
Premature stopping also carries risk. Passing visible tests does not guarantee correctness against hidden tests or real engineering constraints. The objective is therefore not to terminate mechanically at the first green signal. It is to distinguish high-signal, low-cost acceptance checks from unbounded exploration.
Engineering costs and trade-offs
Post-solution steps can consume LLM inference tokens, repository search, build or container resources, CI time, and context-window capacity. A step percentage cannot be converted directly into cost: one large model call may cost far more than a local unit test, while the reverse is also possible.
The central trade-off is between final correctness and efficiency. Stopping immediately could save 35–60% of post-solution steps and eliminate some regressions, but it could also conceal defects that only stronger validation would expose. A better policy looks for the point of diminishing verification returns: after critical regression tests, static checks, and necessary boundary cases pass, uncontrolled exploration should end.
For systems optimizing both correctness and cost, the objective can be approximated as U = P(final_correct) - λ × post_solution_cost - μ × regression_risk. The coefficient λ prices post-solution work, while μ prices incorrect overwrites. Production thresholds should be calibrated from task risk and CI cost rather than imposed as a universal step budget.
Evaluation quality and remaining uncertainty
The strongest methodological feature is the patch-level trace and replay. It treats an agent trajectory as a time series rather than reducing it to one final answer. This makes it possible to distinguish first-correct latency, final correctness, and post-solution behavior. Coverage across five models, two frameworks, and three SWE-bench datasets is also stronger evidence than an isolated model demonstration.
The evidence nevertheless has clear boundaries:
- Correctness depends on the test oracle. Missing hidden tests, flaky tests, or weak coverage can shift the measured time of the first correct patch;
- The main medians cover successful runs and do not directly characterize potentially worse termination lag in failed trajectories;
- Medians conceal long-tail runs that may be far more expensive than the reported 35–60% range;
- The abstract provides no sample sizes, confidence intervals, significance tests, absolute task-success rates, token costs, or wall-clock times;
- The third SWE-bench dataset, the complete model list, and the model-by-framework experimental design are not identified;
- The availability of code, environments, and patch-level logs is not stated.
The study therefore provides strong evidence that post-solution activity is common and potentially harmful. It does not yet establish how much final success rate a particular stopping policy would sacrifice or how much cost it would save.
Deployment guidance for coding-agent frameworks
1. Separate post-solution overhead from total work
Alongside final solve rate and total tokens, record first-correct latency, termination lag in steps and as a percentage, post-solution tokens, test invocations, wall-clock time, regression rate, and the reason for termination. Final accuracy alone cannot distinguish a fast solution from a long path that eventually reaches the same patch.
2. Protect the last known-correct state
After a candidate patch passes acceptance, save an independent checkpoint or Git reference. Run subsequent experiments in isolated branches and automatically restore the last correct commit when a regression is detected, rather than relying on the model to undo its own changes.
3. Use staged stopping conditions
Treat the first correct patch as a trigger for bounded validation rather than an automatic exit signal. This stage should have its own exploration budget, maximum change scope, and repeated-test limit. Normal termination can trigger when the budget is exhausted and high-signal checks continue to pass.
4. Isolate generators from acceptance logic
Tests, static analysis, and regression suites used for acceptance should not be freely editable by the same generated context. The agent may read failure output, but it should not be able to delete assertions or weaken thresholds simply to obtain a passing build. Production systems should also distinguish visible tests from hidden acceptance checks.
5. Calibrate stopping thresholds through replay
Use patch-level logs to replay different stopping thresholds and compare final correctness, post-solution cost, and regression frequency. High-risk repositories may require independent validation or human review before merge, while low-risk and easily reversible tasks can support more aggressive early stopping.
Bottom line
Termination Lag demonstrates that coding-agent inefficiency is not limited to trajectories that fail. Even after a task has passed its tests, agents may spend one-third to nearly two-thirds of their steps on post-solution activity and, in a small fraction of cases, replace a correct result with a broken one. The engineering goal is not merely to produce a patch earlier; it is to identify when the available evidence is sufficient, stop uncontrolled exploration, and preserve the first correct state through checkpoints, isolated validation, and automatic rollback.
— END —Source: Hugging Face Daily Papers (2026-11-11)
Tags: #Coding Agents #SWE-bench #Termination Lag #Agent Stopping #Qwen3-Coder
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments