Multi-Teacher On-Policy Distillation Improves In-Distribution Skills but Struggles with OOD Generalization

Summary:A study exposes a major blind spot in multi-teacher on-policy distillation (OPD/MOPD): student models can nearly match teachers on in-distribution tasks while losing generalization outside the training distribution. Across mathematics, coding, agent, and long-context evaluations, OPD/MOPD score 2.45%–16.25% below the base model on extrapolative OOD tasks and trail the strongest teachers by 18.19%–24.34% on compositional tasks. The authors identify weak teacher capability and insufficient data c

---

Core Finding: Matching Teachers In-Distribution Does Not Mean Transferring Their Generalization

Multi-teacher on-policy distillation (MOPD) attempts to consolidate the capabilities of multiple teachers into a single student. This study identifies a blind spot: a student can nearly match teacher performance on the supervised, in-distribution (IND) domain without inheriting the teachers' out-of-distribution (OOD) generalization.

Using Qwen3 models, the authors evaluate mathematical reasoning, coding, agent capabilities, and long-context performance. The results reveal a consistent pattern of high IND accuracy followed by OOD degradation:

• OPD and MOPD achieve 96.04%–102.73% of their teachers' IND scores.
• On extrapolative OOD tasks, both methods remain 2.45%–16.25% below the base model.
• Across three compositional settings, MOPD improves IND performance but trails the strongest teacher by 18.19%–24.34% on the composed OOD tasks.
• Relative to reinforcement-learning baselines, MOPD reduces OOD forgetting on average, but does not eliminate it.

Teacher accuracy on IND data is therefore an unreliable proxy for successful capability consolidation.

Mechanism: Teacher Signals Cannot Extend Beyond Their Effective Support

Unlike offline distillation, on-policy distillation trains the student on trajectories generated by its current policy. The student is optimized toward the teacher's outputs while its sampling distribution evolves, reducing exposure bias and teacher-student distribution mismatch.

That mechanism does not automatically expand the student's support. In simplified form, the update can be represented as:

[ \theta_{t+1}=\theta_t+\eta g(\text{teacher distribution},\text{student support},D) ]

Here, (g) is the policy-gradient or distillation update and (D) is the training-data distribution. Effective learning occurs only where the teacher's distribution, the student's reachable trajectories, and the available data overlap. A teacher can be highly capable on IND examples without providing a transferable advantage beyond that data boundary.

The study isolates two bottlenecks:

1. Teacher capability: If the teacher lacks the target ability, the student cannot learn it from teacher outputs.
2. Data coverage: Even when the teacher is capable, the student cannot learn the relevant mapping if training trajectories do not cover the required compositions, variations, or long-horizon conditions.

When both bottlenecks are alleviated jointly, OOD forgetting disappears in the experiments, with some evidence of performance beyond the teacher. This suggests that the failure should not be attributed to distillation alone. A strong teacher is useful only when its capability is exposed through data that covers the target behavior.

Update Geometry: Smaller and More Concentrated Updates

The parameter-update analysis shows that OPD and MOPD produce smaller updates than RL baselines, with more concentrated spectral energy. Under MOPD, different teachers also exhibit greater shared update structure.

This geometry helps explain why distillation can yield a compact but narrow policy. Teacher supervision suppresses exploratory noise and concentrates updates on high-confidence behaviors. At the same time, smaller update radii and shared directions reduce the probability that the student discovers alternative strategies outside the teacher's familiar region.

Compression is not generalization. It can faithfully reproduce a teacher's local optimum inside the training distribution while narrowing the policy's behavioral support. RL updates are generally more diffuse and may preserve more exploratory directions, but they can also overwrite existing capabilities. MOPD's average improvement over RL baselines indicates relative conservatism, not a complete solution to OOD retention.

Evaluating the Evidence

The reported gaps are strong enough to reject the assumption that near-teacher IND accuracy proves capability transfer. Nevertheless, several details require examination in the full paper:

• The Qwen3 model sizes, teacher-student combinations, and construction of capability-specific datasets;
• Whether OOD tests are genuinely isolated from teacher training data and free of template or generation-artifact leakage;
• How compositional tasks are defined and whether training budgets are matched across the base model, teachers, and students;
• Variation across random seeds, teachers, and model sizes, including statistical significance;
• Whether performance beyond the teacher is a robust cross-model result or limited to particular OOD metrics.

Proposed Improvements: Geometry Patching and Base-Preserving Replay

The authors introduce two interventions based on the update-geometry analysis.

Geometry-based patching corrects missing or weak directions in the student's update rather than merely adding more optimization steps. In a multi-teacher setting, shared components across teachers can identify capability axes, allowing the method to compensate specifically for underrepresented generalization directions.

Base-preserving replay reintroduces trajectories generated by the base model into training so that distillation does not cause catastrophic forgetting in regions where the base policy remains strong. The replay set must cover examples that the base model handles reliably but teachers may not, rather than duplicating the supervised dataset.

The paper reports that these methods improve OOD performance while largely preserving IND accuracy. The cost is not trivial, however: replay requires additional rollouts, storage, and mixed-objective scheduling, while geometric patching may demand a reliable estimate of update directions and a more complex training pipeline.

Practical Recommendations

Teams using multi-teacher distillation, model merging, or capability-transfer pipelines should revise their acceptance criteria:

1. Stratify evaluation by task composition, input distribution, and difficulty rather than relying only on random train-test splits.
2. Report teacher IND/OOD scores, base-model OOD scores, and student OOD scores separately.
3. Add compositional test sets for mathematics, coding, tool use, and long-context behavior.
4. Add replay from the base or prior policy to the distillation objective and tune its mixture ratio on a dedicated validation set.
5. Inspect update spectra and cross-teacher directional overlap to identify conflicting or overly concentrated updates.

The key deployment criterion is not whether the student reproduces the teacher's IND score. It is whether the student preserves capabilities outside the teacher's training distribution and can compose skills into behaviors that neither the base model nor any individual teacher reliably supports.

---

Source: Hugging Face Daily Papers (2026-11-03)

— END —

Tags: #On-Policy Distillation Follow #Multi-Teacher Distillation Follow #Out-of-Distribution Generalization Follow #Qwen3 Follow #Model Merging Follow