DeepSeek v4.1 Flash Clears the 1% Mark on SWE-Sweep
By Mr.Xu Community Post
Published: · 2 views
Summary:The SWE-Sweep leaderboard has added several more open-weight models, with DeepSeek v4.1 Flash becoming the first open-weight model evaluated above 1% when running in its max reasoning configuration. The benchmark asks models to discover and repair hundreds of hidden bugs in large repository snapshots without being told where the defects are, making it a demanding test of long-horizon reasoning, repository navigation, and tool use. Under the same evaluation harness, Gemini 4 Hydrogen reportedly
Key Result
The community-maintained SWE-Sweep leaderboard has added several more open-weight models. With its max reasoning configuration, DeepSeek v4.1 Flash becomes the first open-weight model evaluated above 1%. The same update lists Gemini 4 Hydrogen at 13.5% and GPT Astra 6 at 9.5%.
The result is worth noting, but it should not be interpreted as evidence that open-weight models have nearly caught up with leading proprietary systems. Crossing 1% is a meaningful first milestone for the open-weight category, while the reported proprietary scores still show a substantial gap. Because the figures come from a community leaderboard rather than a vendor-controlled closed evaluation, the exact values, run-to-run variance, and reproduction cost still need to be verified.
What SWE-Sweep Measures
SWE-Sweep tests whether models can discover and repair hidden defects in real software repositories. A model receives a potentially large repository snapshot containing hundreds of planted bugs, but receives no defect locations, issue types, or reproduction paths. It must search the codebase, invoke build and test tools, interpret failures, modify code, and continue validating its changes.
The benchmark differs from conventional issue-resolution setups in several important ways:
- The defects are completely unknown: Models must decide where to search rather than relying on an issue description that names a file or suspicious function.
- The feedback loop is long: One incorrect diagnosis can trigger a patch, a regression, and another investigation branch, testing long-horizon recovery rather than single-turn generation.
- Repository context is substantial: The model must understand local code while maintaining awareness of cross-directory dependencies, build configuration, and test relationships.
- The score reflects a system-level capability: Results depend not only on the foundation model but also on the agent harness, context window, tool permissions, timeout policy, and sampling budget.
Accordingly, “above 1%” means that, under the current harness and budget, the configuration can solve a small set of difficult long-horizon defects. It is not a linear percentage of ordinary coding ability.
Why a Common Harness Matters
The leaderboard maintainer evaluates all models through the same harness to reduce confounding from agent scaffolding. The maintainer also reports comparisons with Claude Code and Codex harnesses, neither of which appeared to improve results significantly. This suggests that the current gap is driven more by the models themselves than by a specialized coding-agent shell.
A common harness does not, however, eliminate every variable. Models may still receive different reasoning settings, output-token allowances, tool policies, retry budgets, and timeout limits. A max configuration generally implies a larger inference and generation budget, while repository size, test containers, and parallelism can also affect the score. Useful comparisons should record at least:
- The model version and weight hash;
- Maximum context length and total generation budget;
- Available tools and command allowlists;
- Per-task timeouts, retries, and concurrency limits;
- Whether modifications to test files are allowed;
- Success rates and confidence intervals across repeated runs.
Without this metadata, a difference of a few percentage points cannot be attributed cleanly to model architecture or training quality.
Could Multi-Agent Systems Scale Further?
The maintainer suggests that multi-agent systems might expand coverage, for example by assigning separate agents to repository search, patch generation, test analysis, and review. More agents do not automatically produce better accuracy.
Multi-agent designs typically add communication tokens, duplicated context, and coordination overhead. Agents may also share the same erroneous priors, such as misreading an API contract in the same way, producing correlated failures. If an architecture lacks independent verification and adjudication, voting can amplify confident mistakes. A more practical design usually starts with clear role boundaries, test execution, patch review, and rollback-aware state management rather than simply increasing agent count.
Reproduction and Deployment Guidance
The evaluation code is available through GitHub, providing a foundation for independent reproduction. Developers should pin repository snapshots and bug sets, run models in fixed containers, record tool calls, patch diffs, test results, and failure reasons, and repeat the evaluation across multiple random seeds.
For local deployment teams, DeepSeek v4.1 Flash's result demonstrates that open-weight models now show measurable positive signal on long-horizon software repair tasks, rather than being limited to completion, single-function generation, or short patch cycles. At the same time, a score just above 1% means that an automated repair pipeline cannot rely on a single model output. Test gates, static analysis, change review, and rollback mechanisms remain mandatory.
Scaling toward multi-agent execution should begin by measuring marginal benefit. If parallel task coverage grows more slowly than token consumption and wall-clock latency, targeted retrieval and better context organization are likely to offer better cost-efficiency than adding agents without constraints.
Conclusion
DeepSeek v4.1 Flash has crossed the 1% mark on SWE-Sweep, demonstrating that open-weight models can make measurable progress on a difficult benchmark involving hidden defects and long-horizon software maintenance. It still trails the reported scores of Gemini 4 Hydrogen and GPT Astra 6.
The update's broader value is the publicly runnable, unified evaluation path. Only after fixing inference budgets, tool permissions, and test conditions can researchers determine whether the score comes from model capability, agent design, or compute expenditure—and whether it is sufficient for real-world repair systems.
— END —Source: Reddit r/LocalLLaMA (2026-11-13)
Tags: #DeepSeek #SWE-Sweep #Code Repair #Open-Weight Models #Agent Evaluation
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments