GAIR-NLP Open-Sources 801-30B-A3B, Bringing Recursive Self-Improvement to Pre-Training

Summary:GAIR-NLP has released 801-30B-A3B-Base, a mixture-of-experts foundation model with 30 billion total parameters and roughly 3 billion activated parameters per token. The model was trained from scratch on 11.8 trillion tokens over three months. The accompanying paper extends recursive self-improvement beyond post-training by decomposing an unrepeatable frontier pre-training run into smaller, verifiable feedback loops across data, infrastructure, and individual training stages. The project also in

---

Core Release: Moving RSI into Pre-Training

GAIR-NLP has published the paper “No Second Run: Does RSI Stop at Frontier Pre-training?” and plans to open-source 801-30B-A3B-Base together with a lightweight SFT dataset. The work asks a more fundamental question than how to design a stronger RL algorithm: if frontier pre-training consumes hundreds of thousands of GPU-hours over several months and cannot be repeated exactly, can recursive self-improvement still operate?

The proposed answer is not a monolithic loop in which an AI rewrites an entire training run. Instead, it decomposes feedback into smaller, verifiable loops that repair failures in data, infrastructure, and individual training stages. The central idea is that the training production system, rather than only the model, uses diagnostics from previous runs to improve the design and execution of later runs.

Model and Training Scale

801-30B-A3B-Base uses a sparse mixture-of-experts architecture with approximately 30 billion total parameters and about 3 billion activated parameters per token. It was trained from scratch on 11.8 trillion tokens over roughly three months by a newly formed team.

For an MoE model, routing can be represented schematically as:

[ y = \sum_{i \in \mathcal{T}(x)} g_i(x) \cdot E_i(x) ]

where (E_i) is an expert, (g_i(x)) is its routing weight, and (\mathcal{T}(x)) is the selected Top-K expert set. Activating only three billion parameters per token does not make the model operationally trivial: training still incurs routing communication, expert-load imbalance, activation-memory pressure, and large-scale expert parallelism.

The reported post-training phase consists of six billion tokens of lightweight SFT without RL. This is better understood as a probe of the base model’s post-training headroom than as a fully aligned chat or agent model.

Adapting RSI to Unrepeatable Frontier Pre-Training

Conventional post-training RSI benefits from three convenient conditions: an existing base model, affordable trials, and an immediately verifiable objective. Frontier pre-training violates all three.

The method therefore splits the end-to-end process into local loops, such as:

• identifying data-pipeline defects and validating a revised data version;
• adjusting training infrastructure based on throughput, stability, and convergence behavior;
• running controlled ablations or smaller reproductions before deciding what belongs in the next full run;
• feeding post-training failure modes back into data selection and training configuration.

Each local loop can form a diagnose-modify-verify chain. However, the contribution cannot be inferred from the final model score alone. Without concrete examples of detected failures, repaired components, controlled reruns, and ablations, “recursive self-improvement” may amount to a new label for conventional iterative engineering. The paper’s closed-loop cases and attribution evidence are therefore essential.

Reported Results and Evaluation Caveats

The team reports that 801-30B-A3B-Base outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B at the base-model stage, while trailing the larger Qwen3.5-35B-A3B. Its relative position reportedly remains after identical lightweight SFT.

Across four agentic benchmarks, the team also claims that the lightly post-trained model achieves the highest average score against fully post-trained systems, including Qwen3.6-35B-A3B. Three kinds of comparison must be kept separate:

1. base model versus base model;
2. controlled comparison under the same SFT recipe;
3. lightweight SFT versus full post-training.

The third comparison can reveal base-model potential, but it does not demonstrate stronger overall post-training capability. Agentic benchmarks are also sensitive to prompt templates, tool interfaces, decoding settings, scorers, and training-data contamination. The summary does not provide confidence intervals, run-to-run variance, complete baseline configurations, or per-benchmark breakdowns, so the claims should be treated as official results awaiting independent reproduction.

Engineering Trade-offs: Local Feedback, Not Free Retraining

Extending RSI into pre-training can automate improvements across data and infrastructure, but it does not remove the cost of frontier training:

• Training cost: A full run may consume hundreds of thousands of GPU-hours. Even a local validation loop can be expensive if it requires real-model retraining.
• Evaluation noise: Pre-training quality cannot usually be read from a single immediate score. A local proxy may correlate poorly with the eventual model.
• Systems complexity: Data versions, configurations, code changes, and run logs must remain traceable.
• Attribution: The final model emerges from data, architecture, scheduling, and human decisions, making causal attribution difficult.
• Failure containment: Automated repairs touching critical infrastructure require sandboxing, approval gates, and rollback paths.

RSI does not make frontier pre-training cheap. It replaces an intolerable attempt to repeat the entire run with multiple smaller and more rigorous engineering experiments.

Practical Guidance for Developers

The project is currently more relevant to research teams and model engineers than to production users seeking an immediately downloadable checkpoint. The model page is still marked Coming Soon, so developers should initially follow the paper, GitHub repository, and lightweight SFT dataset.

Once weights are available, important checks include:

• expert count, routing Top-K, active parameter count, and checkpoint format;
• memory requirements under BF16, FP8, and quantized formats;
• native support in vLLM, SGLang, TensorRT-LLM, and comparable inference stacks;
• training-data licenses, model license, and commercial-use restrictions;
• reproducible measurements under identical prompts, decoding settings, and tool protocols.

For local deployment, a 30B-A3B model should generally be cheaper than a dense 30B model of the same nominal size, but it should not be described as trivial to run on one consumer GPU. Inference cost still depends on expert routing, memory bandwidth, batch size, and KV Cache length. Long-context agent workloads make KV Cache management particularly important. Hardware recommendations should wait for the official checkpoint and optimized inference implementations.

Conclusion

The primary significance of 801-30B-A3B-Base is not merely another 30B MoE release. It attempts to extend recursive self-improvement from repeatable post-training experiments to the expensive and largely unrepeatable frontier-training system. If controlled evidence shows that local feedback loops reliably produce attributable gains across different failure types, the work could establish a more practical model for automating foundation-model research. Without such evidence, its value remains primarily at the level of training-process innovation.

---

Source: Reddit r/LocalLLaMA (2026-11-15)

— END —

Tags: #801-30B-A3B Follow #Recursive Self-Improvement Follow #MoE Architecture Follow #Frontier Pre-Training Follow #Open Weights Follow