Hugging Face In-Depth Study: Comprehensive Analysis of On-Policy Learning Distillation Dynamics
Summary:Hugging Face's research team conducted a systematic study on the impact of on-policy learning in model distillation. By independently varying rollout policy, token-level KL direction, and learning rate across the Llama3 and Qwen2.5 model families and tasks spanning scientific, medical, and arithmetic domains, the study reveals that rollout policy is not necessarily central. Instead, token-level KL direction more clearly shapes task performance and output coverage, while learning rate governs for
Background and Motivation
On-policy learning has been argued to reduce catastrophic forgetting, produce sparser parameter updates, and improve generalization. However, existing comparisons between supervised fine-tuning and reinforcement learning vary multiple factors simultaneously, making it difficult to isolate the contribution of rollout policy.
Methodology and Findings
Hugging Face's research team conducted a systematic study in a controlled strong-to-weak distillation setting, independently varying rollout policy, token-level KL direction, and learning rate across the Llama3 and Qwen2.5 model families and reasoning tasks spanning scientific, medical, and arithmetic domains. The study reveals that:
- Rollout Policy is Not Central: Rollout policy does not necessarily play a central role in distillation dynamics.
- Token-Level KL Direction Matters: Token-level KL direction more clearly shapes task performance and output coverage. Forward KL is remarkably robust to rollout policy, while reverse KL is more sensitive and favors student-generated rollouts.
- Learning Rate Governs Forgetting and Update Sparsity: The learning rate governs forgetting and update sparsity.
Conclusions and Implications
The findings challenge the conventional view that on-policy rollouts are inherently preferable, showing that their value depends on objectives, evaluation settings, and optimization hyperparameters. This provides new insights into the strategic choices in AI model training.
Recommendations for Developers
- Adjust Strategies Flexibly: In model training, adjust rollout policies flexibly based on specific tasks and objectives, rather than relying on them blindly.
- Focus on KL Direction: Pay attention to the choice of token-level KL direction in the distillation process to optimize task performance.
- Optimize Learning Rate: Set the learning rate reasonably to balance forgetting and update sparsity.
— END —Source: Hugging Face Daily Papers (2026-09-28)
Tags: #Hugging Face #On-Policy Learning #Model Distillation #KL Direction #Learning Rate
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments