ZICQ
中 Log in / Sign up
Newsroom Research & Papers #Multi-Expert Distillation #Reinforcement Learning #Long-Horizon Agents #Off-Policy Distillation #Model Post-Training

MRPD Reuses Expert RL Logs to Consolidate Long-Horizon Agents at One-Seventh the Training Compute

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:Frontier post-training often trains one reinforcement-learning expert per domain, then consolidates those experts through multi-teacher on-policy distillation. This second stage is expensive because the student resamples every environment while teachers score newly generated tokens. MRPD instead reuses each expert’s ordered training log—including rollouts, behavior-policy probabilities, and rewards—and replays those trajectories through the student. The student recomputes policy-gradient updates


Core Idea: Turn Expert Histories into a Consolidation Curriculum

Frontier post-training often follows a two-stage recipe. Teams first train a reinforcement-learning expert for each domain—reasoning, search, SQL, and others—then consolidate those experts into one student through multi-teacher on-policy distillation (MOPD). MOPD is expensive because the student generates fresh trajectories in every environment, multiple teachers score the newly generated tokens, and the student can drift away from the support covered by the final experts.

MRPD starts from a different premise: the expert RL runs have already generated the interaction data needed for consolidation. Those logs contain trajectories, behavior-policy probabilities, rewards, and the chronological order in which the experts learned from them. Rather than resampling the environments, MRPD turns those logs into a curriculum and replays them through the student.

Training Mechanism: Replay Is Not Static Imitation

A useful abstraction of one logged trajectory is τ_e = {(s_t, a_t, log π_beh(a_t|s_t), r_t)}. Here, s_t is the prefix or environment state, a_t is the action or token, π_beh is the historical behavior policy that generated the trajectory, and r_t is the stored reward. A production schema should also preserve timestamps, domains, model checkpoints, reward-function versions, and trajectory boundaries.

During replay, the student recomputes its current action probabilities under the recorded prefixes. A common PPO-style interpretation is:

  • ρ_t(θ) = πθ(a_t|s_t) / π_beh(a_t|s_t), which measures the current student’s divergence from the logged behavior policy;
  • L_clip = -E[min(ρ_t Â_t, clip(ρ_t, 1-ε, 1+ε) Â_t)], which uses logged rewards or advantages derived from them to update the student;
  • L_KD = E[D(πθ(·|s_t) || πφ_e(·|s_t))], which matches the final expert φ_e at the same prefixes;
  • an overall objective conceptually equivalent to L = L_clip + λL_KD.

This is a conceptual reconstruction rather than the paper’s exact formulation. The supplied abstract does not specify the advantage estimator, divergence direction, distillation temperature, clipping coefficient, loss weighting, or implementation details. MRPD is not merely behavior cloning: although actions and rewards are fixed, the student’s changing probabilities alter both gradient magnitudes and action weighting.

Strictly speaking, MRPD remains off-policy because the current student did not generate the replayed trajectories. The paper’s “quasi-on-policy” characterization comes from replaying prefixes that the experts actually visited while PPO-style clipping limits policy displacement. That is an approximation, not an equivalence theorem. Engineers should therefore monitor the fraction of samples reaching the clipping boundaries rather than treating the label as a guarantee.

Where the Compute Savings Come From

MRPD removes online interaction and reward recomputation during consolidation. It does not reset environments, execute search or SQL calls, or query an external reward model. The trainer can read logs, update the student, and query final-expert distributions at stored prefixes within a simpler scheduling path.

The result is not free computation. Student forward and backward passes, value or advantage estimation, final-expert inference, trajectory storage, and replay I/O remain. The reported one-seventh ratio compares consolidation with expert training under the paper’s setup; it does not automatically apply when environments are cheap, teachers are unusually large, or stored logs dominate the cost. MOPD and MRPD should therefore be compared along four separate axes: environment calls, teacher-processed tokens, student-gradient tokens, and wall-clock or accelerator time.

Engineering Trade-offs

| Dimension | Main benefit | Cost or risk to monitor | |---|---|---| | Training cost | Removes environment reruns and online reward calls | Student optimization and final-expert inference remain expensive | | Storage and I/O | One expert run can support repeated consolidation experiments | Token-level trajectories and probabilities grow rapidly; deduplicate prefixes and compress probabilities | | Optimization stability | Expert prefixes are usually closer to feasible trajectories than free student rollouts | Excessive drift causes clipping to suppress learning; track clip fraction, KL, and outlier trajectories | | Coverage | Avoids paying for exploratory sampling during consolidation | Logs cannot cover failure modes the student develops later | | Multi-expert conflict | Ordered replay can reduce interference between domains | Ordering alone does not eliminate catastrophic forgetting; measure cross-domain regressions | | Distillation cost | Final-expert logits can be precomputed and cached | Full-vocabulary KL is memory intensive; Top-K approximations require validation |

Two teacher signals must remain distinct. The historical behavior policy π_beh explains why the logged actions were sampled, while the final expert πφ_e supplies the distillation target. If the final expert has drifted far from the behavior policy, the student must simultaneously satisfy an old-policy gradient constraint and a new-policy distribution target. Rewards can also become stale after changes to reward models, tools, databases, or environment versions. Saving only scalar rewards is therefore insufficient; provenance and versioning are essential.

How to Read the Evaluation

The paper evaluates MRPD on reasoning, search, and SQL, reporting the best student model, substantial gains over online RL baselines, and better performance than MOPD at a fraction of its cost. Comparing MOPD and MRPD with the same experts is a meaningful control because it removes teacher-quality differences as a major confounder. The ablations further indicate that replayed trajectories carry most of the gain and that both the policy-gradient and distillation terms contribute.

The supplied abstract does not report absolute success rates, task counts, expert and student sizes, gradient budgets, seed variance, hardware, or code status. The one-seventh cost and performance claims therefore require reproduction under aligned budgets. Environment interactions, student updates, and teacher inference should be equalized independently. Long-horizon agents also need task success, invalid-action rate, tool-call count, and recovery metrics rather than token-level accuracy alone.

The most consequential finding is that log order matters more than log quantity or reward totals. This suggests that expert histories act as a curriculum rather than an unordered mixture. To isolate the mechanism, experiments should add random-shuffle, reverse-order, reward-sorted, and difficulty-sorted controls under an identical token budget. The authors should also determine whether ordering gains come from better critic fitting, reduced expert interference, or a simpler curriculum-difficulty effect.

Deployment Guidance

  1. Verify prerequisites: Teams need domain experts, access to expert weights and logits, and token-level logs containing behavior probabilities and rewards. API-only access without token distributions is insufficient.
  2. Adopt a versioned log schema: Store trajectory IDs, domains, reconstructable prefixes, action tokens, behavior-policy log probabilities, rewards, values or advantages, model checkpoints, and environment and reward versions.
  3. Build replay as an offline pipeline: Validate completeness, deduplicate prefixes, calibrate probabilities, and only then attach the student optimizer. Avoid changing rewards and curriculum order in the same experiment.
  4. Run layered ablations: Compare MOPD, offline distillation only, policy gradient only, random order, reverse order, and the original learning order while fixing initialization, teachers, token budgets, and decoding settings.
  5. Monitor both optimization and deployment drift: Track clip fraction, per-domain KL, value error, regressions between curriculum stages, and the coverage gap between production trajectories and stored logs.
  6. Retain a refresh path: MRPD is effective for reusing completed RL assets, but it does not replace exploration under changed environments or rewards. A small amount of fresh online data may still be necessary after distribution shifts.

MRPD’s main contribution is not a new foundation model. It turns interaction records—usually discarded after RL training—into reusable post-training assets. Its strongest engineering premise is that environment interaction and reward generation are often more expensive than student optimization, while historical expert trajectories already provide useful local support. Whether that premise becomes a dependable advantage depends on log quality, curriculum design, expert-conflict control, and generalization tests beyond the reported three domains.


Source: Hugging Face Daily Papers (2026-11-18)

— END —

Tags: #Multi-Expert Distillation #Reinforcement Learning #Long-Horizon Agents #Off-Policy Distillation #Model Post-Training

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…