P²-MOPD Accelerates Multi-Teacher On-Policy Distillation with Parameter Priors

Summary:P²-MOPD unifies multi-teacher on-policy distillation with task arithmetic in a single differentiable parameter model. Rather than optimizing student weights and teacher-mixing coefficients separately, the method trains them jointly under the same on-policy distillation objective. Unlike conventional task arithmetic, whose merged model remains within the linear span of frozen student and expert parameters, P²-MOPD keeps the student trainable and treats experts as parameter priors. Across four do

---

Core Idea: Turning Model Merging into a Trainable Distillation Process

P²-MOPD addresses a structural limitation of multi-teacher on-policy distillation (MOPD). Standard MOPD supervises student-generated trajectories with teacher feedback, so expert capabilities enter primarily through the training objective rather than directly through the student's parameter space. Task arithmetic, by contrast, can combine expert weights directly, but typically freezes both the student and experts. The merged model therefore remains constrained to the linear span of fixed parameter tensors.

P²-MOPD represents the student and multiple experts as a differentiable linear combination, then jointly optimizes the mixing coefficients and student weights under one on-policy distillation objective. The important distinction is that this is not a separate merge stage. Mixing coefficients receive feedback from trajectories produced by the student itself, while the student remains trainable and can move beyond the fixed subspace defined by the original parameters.

The method is designed to combine two advantages:

• On-policy distillation: teachers respond to trajectories reachable by the current student, reducing mismatch between offline training distributions and the deployment policy.
• Task-arithmetic parameter priors: teacher weights continuously influence student parameters rather than being applied once as a static merge.

Method Mechanics: Joint Optimization Rather Than Static Combination

A conventional task-arithmetic merge can be expressed as:

$$\theta_{merged}=\theta_s+\sum_i\lambda_i(\theta_i-\theta_s)$$

where the student parameters $\theta_s$, expert parameters $\theta_i$, and mixing coefficients $\lambda_i$ are fixed. No choice of the coefficients can move the result outside the span of those frozen tensors.

P²-MOPD unlocks the student and optimizes both the student weights and the mixing coefficients under the same distillation objective. From an engineering perspective, the mixing coefficients become adaptive control variables rather than post-training merge hyperparameters. A teacher that proves more useful for particular trajectories can receive a larger contribution, while the student can absorb parameter-level priors without remaining confined to the original linear span.

The on-policy aspect is critical. If teachers only label a fixed offline dataset, their apparent advantages may be amplified by distribution coverage. Online distillation asks teachers to provide feedback on states and trajectories currently reachable by the student, which is especially relevant to closed-loop reasoning and control. The cost is substantial: teacher generation or inference must continue during training, increasing memory pressure and compute requirements relative to static task arithmetic.

Results and How to Interpret Them

The paper evaluates P²-MOPD against MOPD across four domains at 1.7B and 4B scales and reports two main findings:

• P²-MOPD reaches MOPD's final average recovery in approximately half the training steps.
• At the end of training, the 1.7B model achieves 94.7% average recovery, compared with 80.9% for MOPD—a difference of 13.8 percentage points.

“Recovery” appears to be a normalized measure of how much student capability is recovered relative to a target or teacher baseline; it should not be interpreted as a universal accuracy metric. Averaging across domains can conceal task-level variance and sensitivity to task mix, normalization, and expert quality. The abstract does not identify the four domains, individual tasks, random seeds, uncertainty, teacher count, or hyperparameter budgets. It is therefore impossible to determine whether the improvement is broadly robust or specific to the reported initialization and metric design.

Training Costs and Engineering Trade-offs

P²-MOPD does not obtain stronger recovery for free. Its main costs are:

1. Multi-teacher inference overhead: teachers must evaluate student trajectories during training. More teachers and longer contexts increase compute and memory demand.
2. Parameter-state overhead: student and teacher weights, coefficients, gradients, and optimizer states may need to coexist. Even a 1.7B model requires careful sharding and recomputation strategies.
3. Coefficient instability: without normalization, constraints, or learning-rate scheduling, coefficients may oscillate between experts or allow one teacher to dominate training.

Compared with static task arithmetic, P²-MOPD is not simpler to deploy; its benefit is that the resulting student can be optimized offline and then run independently. The cost shifts from near-zero training to more expensive distillation. Compared with MOPD, it adds a parameter-prior pathway, which appears to accelerate convergence and balance capability recovery, but the summary does not show whether that improvement survives when total teacher queries and GPU hours are held constant.

Evidence Still Needed

Several details are required before treating the reported gains as production-ready:

• identities and difficulty levels of the four domains;
• number of teachers and the parameterization or constraints on mixing coefficients;
• denominator and aggregation rules for average recovery;
• identical initialization, data, teacher-query budget, and compute for both methods;
• distinction between fewer optimizer steps and lower total cost;
• variance across random seeds and statistical significance;
• availability of weights, configurations, and evaluation code.

The distinction between algorithmic efficiency and wall-clock efficiency is especially important. If P²-MOPD generates substantially more teacher rollouts, its advantage may shrink when measured in GPU hours or teacher tokens rather than training steps.

Practical Deployment Guidance

Teams experimenting with multi-expert consolidation should evaluate P²-MOPD against carefully controlled baselines:

• Hold initialization, experts, data, and effective sample budgets constant, while reporting optimizer steps, GPU hours, and teacher-token costs.
• Normalize or constrain mixing coefficients and track how individual teacher contributions evolve during training.
• Report recovery per task instead of relying only on a cross-domain average.
• Test whether the gains survive long-context, multimodal, and tool-use workloads.
• After training, evaluate whether a final static merge can reduce inference cost without erasing the recovered capability.

P²-MOPD's central contribution is to unify teacher trajectory supervision and teacher parameter priors within one differentiable framework. The reported results suggest meaningful convergence-efficiency potential, but until the teacher configuration, compute accounting, and complete experimental details are available, the method is better viewed as a promising research direction than a universal distillation solution.

---

Source: Hugging Face Daily Papers (2026-11-02)

— END —

Tags: #On-Policy Distillation Follow #Multi-Teacher Distillation Follow #Task Arithmetic Follow #Model Merging Follow #Parameter Priors Follow