Longtail-SWE Opens a Repository-Level Coding Benchmark for 13 Long-Tail Languages

Summary:Longtail-SWE introduces a repository-level software-engineering benchmark with 202 issue-resolution tasks across 13 programming languages that are poorly represented in mainstream evaluations: Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift. Rather than accepting mined issues and tests as ground truth, the benchmark uses a solver–judge–repairer pipeline. A solver attempts each task, while a judge examines the issue, agent trajectory, and test outcomes a

---

Core Release

Longtail-SWE expands software-engineering agent evaluation beyond the Python-dominated GitHub issue landscape. Its main set contains 202 repository-level issue-resolution tasks across Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift.

The project also releases executable environments, complete construction and evaluation tooling, agent trajectories, and an additional set of easier tasks. The main benchmark targets genuine repository-level engineering, while the easier set offers a more accessible evaluation target for smaller models with limited inference budgets.

Why Existing SWE Benchmarks Are Insufficient

Repository-level SWE evaluation presents three distinct engineering problems:

1. Scarce candidate tasks: Many languages have far fewer active repositories, reported issues, and reproducible test suites than Python. Filtering solely by volume excludes entire ecosystems.
2. Tests are not specifications: Tests often encode the behavior of a reference implementation. A test may reject a valid alternative implementation, while incomplete coverage can allow incorrect patches to pass.
3. Ambiguous issue descriptions: Original reports may omit reproduction steps, environment constraints, or expected behavior, making success difficult to evaluate even when the implementation is functionally correct.

Longtail-SWE's main contribution is therefore not merely the number of supported languages. It treats task quality—whether the issue is solvable, the tests are fair, and the description is precise—as a separate engineering problem.

How the Solver–Judge–Repairer Pipeline Works

The benchmark construction follows three stages:

• Solver: An agent attempts to fix the issue inside the target repository environment and records its trajectory, test results, and resource consumption.
• Judge: The judge examines the issue, solver trajectory, and test outcomes to identify ambiguous requirements, faulty tests, and failures that should not be treated as evidence of a bad task.
• Repairer: Revisions are made to tests and issue descriptions to remove tests that reject valid implementations or accept invalid ones, while retaining tasks that remain genuinely challenging.

This approach is more informative than removing every task a model fails to solve. A solver failure can reflect missing language expertise, a difficult build environment, or a genuinely challenging engineering task rather than a defective benchmark item. The judge turns failed trajectories into evidence for repairing specifications and tests.

This design nevertheless introduces bias. A judge may favor languages or implementation styles it understands well, and it may mistake a model's characteristic failure mode for an intrinsic task defect. If revised tests lack independent confirmation that the original behavior should be preserved, the repair process can amplify rather than remove bias.

Evaluation Dimensions and Engineering Costs

The authors evaluate nine model configurations, including frontier models, across task-solving performance, consistency across repeated attempts, and resource usage.

Single-attempt resolution rates can overstate practical reliability because agent sampling, retries, and test-driven iteration introduce substantial run-to-run variance. Repeated-attempt consistency is especially important for deployment: an agent that succeeds occasionally at high inference cost is not equivalent to one that converges reliably within a fixed budget.

Long-tail languages also create different infrastructure costs. Build systems, package managers, compiler versions, and test runners vary across Clojure, Haskell, OCaml, Elixir, and other ecosystems. Reproducible containers do not automatically guarantee stable evaluation; upstream package updates, base-image changes, and platform-specific paths can invalidate previously passing tests. Comparisons should therefore record image digests, dependency locks, timeouts, and complete failure traces rather than reporting resolution rates alone.

Limits of Benchmark Credibility

Although Longtail-SWE goes beyond conventional SWE benchmarks by explicitly reviewing and revising tests and descriptions, an LLM judge is not equivalent to formal verification. Visible repository tests usually represent only a subset of the intended specification. Passing every test does not prove the absence of regressions or full compliance with the issue requirements.

The 202-task main set is also relatively small. Spread across 13 languages, this yields roughly 15 tasks per language, so per-language rankings remain sensitive to task composition. More reliable reporting would include:

• Per-language results with confidence intervals;
• Resolution rates over at least three independent runs;
• The proportion of tasks receiving hidden tests or human review;
• Whether models had access to repository history, repaired tests, or construction tooling;
• The difficulty gap between the main and easier task sets.

If a model improves mainly through larger inference budgets, more sampling attempts, or language-specific scaffolding, the gain should not be attributed directly to general-purpose coding ability.

Practical Deployment Guidance

Longtail-SWE is useful for three primary workflows:

1. Model evaluation: Compare models and agent frameworks outside Python-centric repositories and reduce ecosystem-specific bias.
2. Agent development: Inspect retrieval, editing, testing, and recovery behavior in full trajectories to determine whether failures originate in requirement understanding, patch generation, or tool use.
3. Dataset maintenance: Adapt the solver–judge–repairer process to internal codebases for issue cleanup, reproduction-step generation, and fragile-test repair.

When integrating the benchmark, pin container images and dependencies first, then define language-specific build commands and timeout policies. Evaluation scripts should preserve patches, terminal output, test failure summaries, and per-call token costs. Without those details, a higher resolution rate cannot be reliably attributed to the model rather than the prompt or scaffolding.

For teams seeking evidence of long-tail language capability, Longtail-SWE is more representative of real multi-language software maintenance than a Python-only leaderboard. It is better understood as an extensible evaluation infrastructure than a finished absolute ranking. Its long-term value depends on continued task contributions, stable environment maintenance, and preservation of human validation and historical benchmark versions.

---

Source: Hugging Face Daily Papers (2026-11-06)

— END —

Tags: #Coding Agents Follow #SWE Benchmarks Follow #Multilingual Programming Follow #Repository-Level Evaluation Follow #Benchmark Quality Control Follow