ZICQ
中 Log in / Sign up
Newsroom Research & Papers #Hugging Face #On-Policy Learning #Model Distillation #KL Direction #Learning Rate

Hugging Face In-Depth Study: Comprehensive Analysis of On-Policy Learning Distillation Dynamics

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 3 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:Hugging Face's research team conducted a systematic study on the impact of on-policy learning in model distillation. By independently varying rollout policy, token-level KL direction, and learning rate across the Llama3 and Qwen2.5 model families and tasks spanning scientific, medical, and arithmetic domains, the study reveals that rollout policy is not necessarily central. Instead, token-level KL direction more clearly shapes task performance and output coverage, while learning rate governs for


Background and Motivation

On-policy learning has been argued to reduce catastrophic forgetting, produce sparser parameter updates, and improve generalization. However, existing comparisons between supervised fine-tuning and reinforcement learning vary multiple factors simultaneously, making it difficult to isolate the contribution of rollout policy.

Methodology and Findings

Hugging Face's research team conducted a systematic study in a controlled strong-to-weak distillation setting, independently varying rollout policy, token-level KL direction, and learning rate across the Llama3 and Qwen2.5 model families and reasoning tasks spanning scientific, medical, and arithmetic domains. The study reveals that:

  1. Rollout Policy is Not Central: Rollout policy does not necessarily play a central role in distillation dynamics.
  2. Token-Level KL Direction Matters: Token-level KL direction more clearly shapes task performance and output coverage. Forward KL is remarkably robust to rollout policy, while reverse KL is more sensitive and favors student-generated rollouts.
  3. Learning Rate Governs Forgetting and Update Sparsity: The learning rate governs forgetting and update sparsity.

Conclusions and Implications

The findings challenge the conventional view that on-policy rollouts are inherently preferable, showing that their value depends on objectives, evaluation settings, and optimization hyperparameters. This provides new insights into the strategic choices in AI model training.

Recommendations for Developers

  1. Adjust Strategies Flexibly: In model training, adjust rollout policies flexibly based on specific tasks and objectives, rather than relying on them blindly.
  2. Focus on KL Direction: Pay attention to the choice of token-level KL direction in the distillation process to optimize task performance.
  3. Optimize Learning Rate: Set the learning rate reasonably to balance forgetting and update sparsity.

Source: Hugging Face Daily Papers (2026-09-28)

— END —

Tags: #Hugging Face #On-Policy Learning #Model Distillation #KL Direction #Learning Rate

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…