BuggyMLE Scales Harder Repository-Level MLE Benchmarks with CCI
Summary:BuggyMLE reframes machine-learning engineering evaluation as a repository-scale agent problem rather than a set of single-file edits. Its Code Complexity Index, or CCI, combines ten structural signals spanning repository size, dependency depth, and patch scope. The index predicts average solve rates across eight public coding benchmarks with an R² of 0.951 and guides a mutation pipeline that turns 222 source repositories into more than 6,600 faulty variants containing nearly 260,000 injected lin
Core Result
BuggyMLE expands machine-learning engineering evaluation from editing isolated snippets to operating inside complete repositories: reproducing experiments, debugging training runs, integrating data, and porting frameworks. Its main contribution is not a new model architecture, but an extensible fault-synthesis pipeline, the Code Complexity Index, or CCI, and a difficulty-graded repository benchmark called BuggyMLE-Bench-Lite.
| Component | Reported scale | Purpose | |---|---:|---| | CCI | Ten structural signals | Predict average solve rate for repository tasks | | BuggyMLE generator | 222 source repositories | Inject controlled faults into working repositories | | Synthetic task set | More than 6,600 variants and nearly 260,000 injected lines | Provide a scalable pool of MLE tasks | | BuggyMLE-Bench-Lite | Three difficulty tiers | Test multi-file editing, environment operation, and debugging | | Training curriculum | 200 additional repositories across three tiers | Fine-tune a Qwen3.5-9B-based model |
Mechanism: Structuring Repository Difficulty
Existing evaluations such as MLE-Bench Lite often restrict changes to a single file and withhold full repository or shell access. That reduces environment cost, but it also removes the hardest parts of real machine-learning engineering: understanding project structure, tracing cross-file dependencies, running training commands, interpreting logs, and checking whether a change breaks unrelated modules. BuggyMLE restores this full execution surface.
CCI can be represented abstractly as CCI = f(x₁,...,x₁₀), with the ten inputs spanning repository scale, dependency depth, and patch scope. Deeper dependency graphs expand the number of interfaces an agent must retrieve and reconcile. Larger patches usually imply more files, configuration changes, and potential side effects. The paper reports that CCI predicts average solve rates across eight public coding benchmarks with an R² of 0.951, with patch size and complexity emerging as the strongest predictors.
This suggests that repository difficulty is not a single size axis. Two repositories with similar line counts can impose very different agent costs because their dependency graphs, build chains, and test entry points differ. If CCI is calibrated across more model families, repository types, and tool configurations, it could support task stratification, capability diagnosis, curriculum sampling, and cost-aware model routing.
The abstract does not disclose the exact definition and normalization of all ten features, the regression model, or its coefficients. The high R² therefore demonstrates predictive association rather than causality. Patch size may partly proxy difficulty, test leakage, or dataset artifacts. CCI is better treated as an empirical difficulty coordinate than as a universal and fully interpretable scalar.
Synthesizing Faults from Working Repositories
BuggyMLE avoids waiting for real projects to organically accumulate enough high-quality failures. Its generation pipeline can be understood as four stages: select a machine-learning repository with a working build and test flow; choose an injection point tied to a specific function; generate a controlled mutation that preserves basic executability; and retain only instances that pass build, unit-test, and task-checker validation.
The main advantage is scalable difficulty with executable environments and reference patches. The cost is distributional mismatch with production incidents. Syntax errors are easy to generate, while configuration drift, implicit state coupling, distributed-training instability, and cross-version framework incompatibilities are closer to real MLE but much harder to synthesize. If mutation operators repeatedly leave recognizable traces, an agent may learn to identify those traces rather than perform general root-cause analysis.
Whether the generator includes genuine issues, whether it stratifies fault types, and whether build or test checkers reveal the root cause are therefore critical. The reported repository count and injected-line volume demonstrate scalability, but not that synthetic defects are statistically equivalent to defects observed in production.
Benchmark Difficulty and Transfer Results
The paper evaluates 14 models on BuggyMLE-Bench-Lite. The best result at each difficulty tier falls from 95% on the easiest tier to 9% on the hardest, a difference of 86 percentage points. This sharply reduces score saturation caused by shallow or single-file tasks and indicates that repository navigation, shell operation, and multi-file debugging expose substantial weaknesses in sustained planning, tool use, and error recovery.
The 95% to 9% change, however, represents the best score at each tier rather than a longitudinal decline by one model across all tiers. The abstract does not provide the complete 14-model matrix, confidence intervals, runtime budgets, or failure taxonomy. It is therefore impossible to isolate whether the decline comes from code reasoning, repository navigation, dependency installation, training timeouts, or differences in agent scaffolding.
The researchers also construct a three-tier curriculum from 200 additional machine-learning repositories and fine-tune a model based on Qwen3.5-9B. The resulting model reaches the paper's reported frontier-level competition on BuggyMLE-Bench-Lite and transfers to general software engineering: SWE-bench Lite resolve rate rises from 24% to 33%, while LiveCodeBench v6 pass@1 increases from 69% to 71%. The nine-point SWE-bench gain suggests that repository-level training may improve cross-file debugging and tool coordination. The two-point LiveCodeBench gain is smaller, and the absence of seed variance, training compute, or an equally sized continued-pretraining baseline prevents attributing all improvement to the fault-injection curriculum.
Engineering Trade-offs and Evaluation Risks
- Controllability versus realism: Programmatic mutations provide stable difficulty tiers and fault categories, but synthetic defects may contain artificial signatures. Real issues are more natural but harder to reproduce and label.
- Elevated contamination risk: Source repositories, tests, and curriculum data come from public code. If a repository or patch appeared in pretraining, a model may locate an answer from memory rather than perform repository analysis.
- Nontrivial infrastructure cost: Six thousand tasks do not mean that only a small amount of text must be copied. Dependency downloads, container images, GPU scheduling, and caching magnify evaluation cost, and environment failures can be mistaken for coding failures.
- Tool budgets can confound difficulty: Shell access, networking, and timeout policies can independently alter solve rates. Evaluations should fix tool sets, context budgets, command timeouts, and retry policies.
- CCI still needs external validation: A high R² may be dominated by coarse variables such as patch size. Cross-validation should cover different models, repository types, and unseen benchmarks, with per-task prediction errors reported.
- Fine-tuning gains need stronger controls: Beyond the original Qwen3.5-9B scores, comparisons should include general code corpora, an equal volume of real issues, ordinary software-repair data, and a no-curriculum control group.
Developer Adoption and Deployment
- Build an isolated repository runner: Give every task its own container or microVM, a writable workspace, frozen dependencies, and fixed test commands. Restrict network access, CPU, GPU, memory, and wall-clock time, and prevent the agent from modifying the evaluator or hidden tests.
- Record complete trajectories: In addition to final solve rate, capture retrieved files, executed commands, test-failure categories, patch rollbacks, and token consumption. This separates reasoning failures from infrastructure failures.
- Adopt tiered reporting: Stratify tasks by CCI into easy, medium, and hard groups, then report pass@1, mean repair steps, cost, and confidence intervals for each tier instead of relying on one aggregate score.
- Enforce decontamination: Apply temporal splits and similarity search across repository contents, patches, test strings, and issue descriptions. Keep training repositories separate and evaluate with held-out mutation operators or real defects.
- Retain a real-fault mixture: Combine synthetic bugs, real issues, and no-fault tasks so the agent does not learn that every failure has a predictable injection pattern. Report results separately by fault category.
- Enforce permission boundaries in production: When applying the approach to real MLE agents, separate permissions for diagnosis, source editing, data writes, and training launches. Add human approval or policy gates for configuration changes, data overwrites, and compute consumption.
Conclusion
BuggyMLE's main contribution is turning increasingly saturated repository-level coding evaluation into a research infrastructure that can scale along a controlled difficulty axis. CCI, the fault synthesizer, the graded benchmark, and curriculum training form a coherent loop. Yet the current evidence is still limited to an arXiv v1 abstract. Independent validation of synthetic-fault realism, decontamination, complete score distributions, and code reproducibility is necessary before the benchmark can serve as a strong model-ranking authority.
— END —Source: ArXiv AI (cs.AI) (2026-11-20)
Tags: #BuggyMLE #MLE Agents #Repository-Level Coding Agents #Code Complexity Index #Qwen3.5-9B
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments