M3OPD Open-Sources Multi-Teacher On-Policy Distillation for 9B Multimodal Reasoning
Summary:M3OPD-Reasoner presents an open post-training recipe for vision-language models. Starting from Qwen3.5-9B, it trains separate GSPO teachers for image STEM, image perception, and video, then merges their capabilities into one student through multi-teacher on-policy distillation. The release includes 1.35 million SFT examples, 193,000 RLVR examples, training code, and checkpoints from every stage. The reported best sequence applies MixRL after MOPD using the same prompts. The authors report that
Core Result
M3OPD-Reasoner provides a reproducible post-training pipeline for vision-language models. Starting from Qwen3.5-9B, it trains separate teachers for image STEM, image perception, and video reasoning with Group Sequence Policy Optimization. Multi-Teacher On-Policy Distillation then merges those specialized capabilities into one student, after which MixRL is applied to produce the final M3OPD-Reasoner-9B checkpoint. The project also releases M3OPD-SFT-1.35M, M3OPD-RL-193K, training code, and checkpoints from every stage.
The main contribution is therefore not a larger backbone model, but a way to reuse domain-expert training, data, and optimization compute across sequential post-training stages. The reported best recipe trains the teachers directly with GSPO, without an additional SFT warm-up, distills them into a student, and then runs MixRL on the same prompts.
How the Recipe Works
- Train domain teachers: Three GSPO teachers are optimized from Qwen3.5-9B for image STEM, image perception, and video reasoning.
- Distill on the student policy: The student generates its own response, and the relevant domain teacher supplies a learning signal for that student-produced trajectory. Training therefore follows the student's current policy rather than imitating only static teacher answers.
- Continue with MixRL: The MOPD student has already merged capabilities from all three domains and becomes a stronger starting policy. MixRL uses the same prompts, but the updated student generates a new on-policy distribution instead of replaying the old trajectories.
- Publish stage checkpoints: The release retains the specialist teachers, the MOPD student, the final model, and intermediate training artifacts.
Mechanism: Why On-Policy Distillation Matters
Static SFT trains on a fixed collection of high-quality answers. Once the deployed model deviates from those demonstrations, however, the training distribution and the actual generation distribution separate. MOPD lets the student generate under its current policy and then asks the appropriate teacher to correct those trajectories. More gradient mass is consequently spent on states where the student currently makes mistakes.
Conceptually, the objective can be represented as:
L_OPD = E[x, y ~ pi_student][w(x) * L(pi_student, teacher_signal)]
Here, w(x) selects the domain teacher, while teacher_signal represents a score or distribution constraint over the student's trajectory. The available abstract does not specify whether the final loss operates at token level, sequence level, or a mixture, nor does it disclose the teacher-signal aggregation weights. The released implementation is therefore essential for reproducing loss scaling, long-sequence bias, and teacher-call cost.
GSPO approximates the policy ratio at the sequence level and applies sequence-level advantages to token gradients. This avoids mixing a sequence-level outcome reward with token-level importance ratios that may follow very different scales. That distinction matters for video and long reasoning traces, where one faulty decision may emerge hundreds of tokens after the prompt.
Running MOPD and MixRL consecutively also creates an important distinction: the prompts remain the same, but MOPD changes the student policy. MixRL consequently samples a new response distribution rather than merely replaying old outputs. This explains how the same prompts can still provide useful learning signals, although it also creates a risk of overfitting to fixed prompts and verifiable rewards.
Engineering Trade-offs
- Training costs more than ordinary SFT: The system maintains three specialist teachers and repeatedly performs student rollout, teacher scoring, and backpropagation. Video examples further amplify visual-token count, decoding latency, and memory pressure. Reproducing 1.35 million SFT examples plus 193,000 RLVR examples is a multi-GPU engineering task rather than a single-device experiment.
- Serving may become simpler: Inference requires one 9B student instead of keeping three teachers resident or routing every request through a teacher ensemble. If the unified model shows material weakness outside its strongest domain, request-level routing to one specialist remains an alternative.
- Memory cannot be estimated from weights alone: BF16 weights alone require roughly 18 GB in decimal units. The deployment must also accommodate the vision encoder, intermediate activations, and KV Cache. A useful approximation is
2 × L × H_kv × d_head × T × B × bytes_per_element, where video frames, resolution, concurrency, and grouped-query attention all affect the result. - Teacher calls create a cost curve: On-policy distillation avoids having teachers generate full demonstrations, but teachers must still score or evaluate long student responses. Longer reasoning chains therefore increase teacher forward passes and KV-cache management overhead.
- Direct GSPO may not remain optimal everywhere: The paper identifies directly trained GSPO teachers as its best recipe, but different vision encoders, video sampling policies, and reward models could change that ranking. An SFT warm-up adds cost but may still improve optimization stability.
How to Read the Benchmarks
The study compares MixSFT, MixRL, specialist teachers, and MOPD across 21 image STEM, image perception, and video reasoning benchmarks. It also decontaminates training data against the evaluation sets, which is more rigorous than reporting leaderboard scores alone. The authors report that the MOPD student approaches each teacher on its own domain, outperforms mixed-training baselines on average, and continues improving through MixRL without collapse.
The available record does not include absolute scores, per-domain breakdowns, variance across seeds, decoding settings, generation budgets, or teacher inference costs. Consequently, matching teachers and beating baselines on average do not prove that all three domains were preserved equally. A stronger evaluation would publish the full teacher-domain × student-domain × evaluation-domain matrix and highlight worst-domain performance rather than relying only on cross-task averages. If video scores rely on an automatic judge, human audits and judge-consistency measurements are also necessary.
Deployment Guidance
- Reproduce the stages incrementally: Starting from Qwen3.5-9B, evaluate the three GSPO teachers, the MOPD student, and the post-MixRL model separately. Record GPU hours, rollout tokens, teacher calls, and peak memory at every stage. Under a constrained compute budget, compare the MOPD student directly with the final checkpoint.
- Set a production visual budget: Fix frame count, image resolution, and maximum generation length, then measure throughput, time to first token, KV-cache occupancy, and long-video degradation. Short public benchmarks are poor proxies for production video workloads.
- Use an identical evaluation protocol: Mixed training, the MOPD student, and the final model should receive the same prompts, temperatures, token limits, and tool permissions. Run multiple seeds per domain and publish per-example results.
- Audit the release licenses: Although the data, code, and checkpoints are described as open, commercial users must verify code and model licenses, dataset provenance, redistribution rights, and restrictions on image or video material.
- Test whether unification is worthwhile: Compare the single M3OPD-Reasoner-9B deployment against a router that selects one specialist teacher per request. If the unified model provides only a small quality gain but materially increases multimodal latency, domain routing may offer better operational economics. If it performs consistently, one model simplifies scaling, monitoring, and failover.
Boundaries of the Evidence
M3OPD-Reasoner demonstrates the feasibility of multi-teacher on-policy distillation for vision-language models and unusually provides data, code, and intermediate checkpoints. The current evidence supports further reproduction more strongly than a universal claim of final-model leadership. Absolute metrics, training cost, the exact teacher-signal formulation, licensing, and independent replication remain the decisive questions for production adoption.
— END —Source: Hugging Face Daily Papers (2026-11-17)
Tags: #M3OPD-Reasoner #Multimodal Reasoning #On-Policy Distillation #GSPO #Reinforcement Learning
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments