ZICQ
中 Log in / Sign up
Newsroom Agentic #RepoFactory #RepoHorizon #Coding Agents #Environment Synthesis #Long-Horizon Training

RepoFactory Evolves Verifiable Coding Environments, Boosting RepoHorizon by 22.6 Points

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:RepoFactory converts real software repositories into 10,159 verifiable training environments spanning 17 domains. Agents receive a project specification and must build a complete implementation from scratch, while an Environment Evolution Loop uses proxy feedback to optimize validity, structural coupling, and implementation scope. Training Qwen3.5-35B-A3B on these environments produces RepoHorizon-35B-A3B. The paper reports a 22.6-percentage-point gain on NL2Repo-Bench and an 8.5-point average


Core Result

RepoFactory does not introduce a new inference architecture. Its focus is the training supply chain for coding agents: real repositories are converted into verifiable environments in which an agent must build a complete project from a specification, and an Environment Evolution Loop continuously improves the environments through proxy feedback.

The paper reports 10,159 environments spanning 17 software domains. Training Qwen3.5-35B-A3B on them produces RepoHorizon-35B-A3B, with a reported 22.6-percentage-point gain on NL2Repo-Bench and an 8.5-point average gain across five long-horizon coding and agentic benchmarks.

Technical Mechanism

From repositories to verifiable environments

The abstract establishes the functional pipeline but does not disclose the repository parser, test framework, or extraction algorithm. At a minimum, RepoFactory must perform three transformations:

  1. Repository to specification: Condense observable functionality, interfaces, and dependencies from a real repository into a task specification without exposing the original implementation as the answer.
  2. Specification to engineering environment: Give the agent the directories, dependencies, and execution entry points required to implement an entire project from scratch rather than a single function or isolated snippet.
  3. Artifact to proxy feedback: Return a reproducible evaluation signal covering validity, component coupling, and implementation scope. The abstract does not state whether the verifier uses compilation, unit tests, integration tests, or behavioral tests, which is critical to reproducibility.

Multiplicative environment objective

The reported objective can be abstracted as:

J(E) = V(E) × C(E) × S(E)

Here, V represents validity, C structural coupling, and S implementation scope. Their normalization and weighting are not disclosed. Multiplication creates a gating effect: a near-zero value in one dimension collapses the overall score, discouraging tasks that merely compile, produce empty directory structures, or contain disconnected components.

The trade-off is a sparser and more sensitive optimization signal. If the dimensions use incompatible scales, one metric may dominate candidate ranking indefinitely. The generator may also discover shortcuts that satisfy superficial structure without genuine functional coupling. The environment score therefore needs hidden tests, behavioral checks, and out-of-distribution environments rather than a single public verifier that agents can game.

Proxy feedback for faster recursive improvement

Conventional recursive self-improvement often follows a serial loop of model rollout, model training, downstream evaluation, and another rollout. Long-horizon coding requires cross-file edits, repeated tool calls, and iterative repair, making each downstream evaluation expensive and slow. RepoFactory moves environment selection earlier and uses faster proxy signals to prioritize environments with greater expected training value.

This does not remove the cost of final evaluation. It decouples environment selection from expensive model training, but any mismatch between proxy metrics and the target task distribution will systematically bias training toward environments that are easy to game.

Engineering Trade-offs

The verifier sets the training ceiling

Producing 10,159 environments does not guarantee that all of them carry equal training value. If verification checks only file existence, interface signatures, or public tests, agents can generate compilable yet behaviorally incorrect projects. Overly strict verification, meanwhile, rejects valid intermediate states. A robust setup should layer formatting and compilation checks, module-level tests, and hidden end-to-end behavioral tests while recording the failure category for every environment.

Multiplicative objectives are sparse and ranking-sensitive

The multiplicative form prevents one excellent dimension from hiding a fatal defect, but it may also leave most candidate environments near zero and concentrate ranking differences in a small subset. Operators should log the joint distribution of V, C, and S, along with zero-score rates, score variance, and rejection ratios. Extreme-score domination may indicate unstable normalization or truncation rather than genuine environment quality.

Scale does not close the compute-cost equation

The abstract does not report repository parsing time, container build cost, verifier runtime, rollout tokens, GPU type, parallelism, or total training expenditure. Coding-agent training is often constrained by sandbox startup, test execution, and tool latency rather than model FLOPs alone; idle GPUs waiting for slow validators can dominate the wall-clock budget.

Long trajectories increase serving overhead

Long-horizon tasks accumulate source code, tool output, and error feedback. For a conventional Transformer decoder, KV-cache memory for one sequence is approximately:

2 × L_layers × L_seq × H_kv × d_head × bytes_per_element

Batch size multiplies this footprint. GQA or MQA reduces H_kv relative to the number of query heads. RepoHorizon's complete configuration is not provided. The 35B-A3B name conventionally suggests a MoE model with roughly 35B total parameters and about 3B active parameters per token, but this must be confirmed from the model card. If it is MoE, all expert weights still occupy storage or distributed deployment, and expert routing and inter-device communication affect throughput; A3B does not mean the full model needs only 3B parameters of memory.

Reported Results

| Training condition | Reported NL2Repo-Bench gain | Reported average gain across five benchmarks | |---|---:|---:| | RepoFactory environments with environment evolution | 22.6 percentage points | 8.5 points | | Without environment evolution | 17.5 percentage points | 6.4 points | | Arithmetic difference from the evolution loop | 5.1 percentage points | 2.1 points |

Under the abstract's comparison, large-scale verifiable environments provide most of the gain, while the evolution loop adds further improvement. However, because the paper does not fully establish whether environment count, rollout tokens, training steps, and inference budgets were matched, the 5.1- and 2.1-point differences should not yet be treated as strict causal estimates.

Important evaluation details are also missing. The paper reports relative gains but not absolute scores, baseline results, random seeds, variance, or statistical significance. The composition of the five benchmarks, the provenance of production repositories, and their difficulty distribution are unspecified. Because both the generated environments and NL2Repo-Bench involve repositories, lineage deduplication, temporal splits, and license checks are essential to rule out contamination.

Reproduction and Deployment

  1. Match training budgets: When comparing random environments, fixed-quality selection, and environment evolution, hold the base checkpoint, environment count, rollout tokens, optimization steps, and inference budget constant.
  2. Keep hidden verifiers isolated: Use public checks only for training feedback and evaluate with hidden tests to discourage verifier-specific optimization.
  3. Perform repository-level deduplication: Link tasks through repository URLs, Git history, code fragments, and license families rather than filenames or prompt strings alone.
  4. Monitor the full objective: Track each multiplicative component, the zero-score rate, and ranking stability across rounds to detect objective collapse.
  5. Separate capability from safety: Execute generated code in sandboxes without default credentials, with restricted networking, CPU and memory quotas, timeouts, and artifact cleanup.
  6. Verify artifact availability: The abstract does not confirm whether code, environments, training scripts, or RepoHorizon weights are released. Without them, RepoFactory is best treated as a research direction for environment engineering rather than a drop-in RL component.

Open Questions

The abstract does not identify the authors' affiliation, absolute benchmark scores, training cost, hardware, variance, contamination controls, full ablations, or licensing status. It also leaves open whether RepoHorizon weights and the generated environments are publicly available, as well as whether third-party proprietary code entered the synthesis pipeline.

The main contribution is therefore a plausible answer to a key coding-agent bottleneck: performance depends not only on model scale and reinforcement-learning algorithms, but also on whether teams can continuously generate realistic, structurally coherent, verifiable, and appropriately difficult long-horizon environments. Whether the evolution loop remains stable at larger scale, and whether its proxy objective creates verifier exploitation, still requires code, weights, and independent replication.


Source: Hugging Face Daily Papers (2026-11-17)

— END —

Tags: #RepoFactory #RepoHorizon #Coding Agents #Environment Synthesis #Long-Horizon Training

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…