Pass_X@k Shows Debuggers Do Not Outperform Two Bash Rollouts

Summary:The paper isolates the effect of an interactive debugger by holding the coding-agent scaffold fixed and comparing a debugger-only agent with a bash-only agent across all 500 SWE-bench Verified instances. The evaluation uses GPT-OSS-120B and Nemotron-3-Ultra-550B, with three independent runs for every model-and-interface configuration. To separate genuine tool complementarity from variance introduced by repeated sampling, the authors introduce pass_X@k, a cross-agent estimator of the probability

---

Core Finding

With the model, task set, and agent scaffold held fixed, the researchers directly compared two software-engineering agents: a debugger-only agent that inspects runtime state through an interactive debugger, and a bash-only agent that relies on an edit-execute-observe loop. The experiment covered all 500 SWE-bench Verified instances, with three independent runs for each model-and-interface configuration using GPT-OSS-120B and Nemotron-3-Ultra-550B.

The results do not support the assumption that adding a debugger automatically produces a stable improvement. For both models, two bash rollouts achieved a higher combined success rate than one bash rollout plus one debugger rollout.

Why Single-Run Results Can Create a False Tool-Complementarity Signal

Coding agents are inherently stochastic. Even when the model, prompts, permissions, and task set remain identical, different rollouts may solve different instances because of sampling variation, divergent exploration paths, or intermediate reasoning forks. As a result, the union of any two runs often appears to show complementary behavior:

• The bash agent solves task A.
• The debugger agent solves task B.
• Their union appears to cover more of the benchmark.

This only demonstrates that two random samples produced partially disjoint solution sets. It does not establish stable specialization between the debugger and bash. Testing genuine complementarity requires estimating the probability that at least one sampled agent succeeds, rather than simply comparing the union from one pair of runs.

Statistical Mechanism Behind pass_X@k

The paper extends the cross-agent pass@k idea into pass_X@K: sample K independent rollouts from each configuration and estimate the probability that at least one of them succeeds. The important distinction is that different tools are not treated as a fixed ensemble; instead, different rollout combinations can be compared under a common budget.

For a success probability of p, the probability that at least one of K independent rollouts succeeds is:

p_X = 1 - (1 - p)^K

Under the independent and identically distributed sampling assumption, pass_X@K increases with K because repeated sampling captures more of the upper tail of the success distribution. It does not, by itself, establish that the tools possess complementary capabilities. In this experiment, “one bash plus one debugger” is a heterogeneous sample with a budget of two, whereas “two bash rollouts” is a homogeneous sample with the same budget. If the former does not outperform the latter, partially disjoint solution sets cannot justify a claim of debugger complementarity.

The framework also implies that tool combinations should be evaluated under equal rollout budgets, model choices, and closely matched scaffolds. Average success alone, or the union observed in a single pair, is insufficient.

Potential Value and Engineering Boundaries of Debuggers

An interactive debugger provides direct access to breakpoints, stack frames, local variables, and runtime control flow. In theory, it should be more useful than repeated shell commands for diagnosing concurrency bugs, complex state transitions, and failures buried in deep call chains. It can also reduce context noise caused by blind code changes and repeated print statements.

Those capabilities do not automatically translate into a higher one-shot success rate:

1. The tool does not provide reasoning by itself: the model must still decide where to place breakpoints, which variables to inspect, and how to turn runtime evidence into a correct patch.
2. Interactive overhead is higher: every breakpoint operation may add command round trips and token consumption, reducing the effective context available to the model.
3. Environment adaptation costs are substantial: debugger commands differ across languages, build systems, and test frameworks, while a general bash interface usually has lower integration friction.
4. Scaffold effects can contaminate evaluation: changing prompts, subagents, tool wrappers, and retry policies at the same time makes it impossible to attribute gains to the debugger itself.

Evaluation Strengths and Limitations

The study's main strength is strict experimental isolation: it covers the complete 500-task benchmark rather than a selected subset; it performs three independent runs for every model-and-architecture configuration; and it holds the remaining scaffold fixed. This design is more informative than adding a debugger while simultaneously changing the agent framework.

However, the available abstract does not provide complete confidence intervals, per-model success-rate differences, token costs, or debugger invocation costs. Even though pass_X@2 points in the same direction for both models, three independent runs are not enough to characterize the variance distribution precisely across all tasks. External validity may also depend on how many SWE-bench tasks actually require native debugger features, runtime differences across languages, and the distribution of failure modes.

The cross-agent estimator additionally relies on the assumption that rollouts are approximately independent and identically distributed. Shared context contamination, repository caches, or side effects in the test environment could violate that assumption. Production evaluations should therefore report success under an equal budget, token consumption, end-to-end latency, and human-intervention rate together.

Implications for Coding-Agent Development

A debugger remains useful as a pluggable capability, but it should not automatically replace a mature bash workflow. A more defensible design is task-aware routing:

• Use bash and test scripts by default for ordinary build failures, test assertions, and local logic errors.
• Enable breakpoint-based runtime inspection for concurrency races, state-machine anomalies, and failures in deep call chains.
• When the budget is fixed, allocate remaining samples according to expected information gain rather than adding a superficially different tool.
• Use cross-agent success probabilities to evaluate tool routing, avoiding the mistake of treating sampling variance as professional specialization.

For agent-evaluation platforms, pass_X@K provides a stricter criterion for tool diversity than a simple solution-set union. It does not prove that debuggers are useless. Instead, it requires developers to establish how much net benefit an heterogeneous tool mix delivers relative to repeated bash sampling under the same budget, and whether that benefit exceeds the additional latency and token cost.

---

Source: ArXiv AI (cs.AI) (2026-11-10)

— END —

Tags: #Coding Agents Follow #pass_X@k Follow #SWE-bench Follow #Interactive Debugger Follow #Agent Evaluation Follow