Hugging Face Proposes New OPD Method Based on Relative Policy Differences: Revolutionizing Reinforcement Learning Agent

Summary:Hugging Face's research team introduces a novel approach to On-Policy Distillation (OPD) based on relative policy differences. This method defines an implicit reward through the log-probability ratio between a proxy expert and a proxy reference policy, enabling efficient knowledge transfer across models of different scales, training stages, or generations. Experiments demonstrate that this approach excels in task-specific training, such as reinforcement learning and supervised fine-tuning, while

---

1. Background and Motivation

On-Policy Distillation (OPD) is a knowledge transfer method in reinforcement learning that trains a target policy using token-level supervision from a stronger teacher policy. However, traditional OPD methods rely on the teacher policy being significantly superior to the target, which may not always be feasible in practice. To address this limitation, Hugging Face's research team proposes a new OPD approach based on relative policy differences, where the log-probability ratio between a proxy expert and a proxy reference policy defines an implicit reward for KL-regularized policy optimization, enabling more flexible knowledge transfer.

2. Methodology and Core Mechanisms

The core of this method lies in transforming policy differences into log-probability ratios and using them as an implicit reward for KL-regularized policy optimization. Specifically, the log-probability ratio between the proxy expert and the proxy reference policy defines a reward function that guides the target policy towards the expert while filtering out shared behaviors through the reference policy, retaining relevant residuals. This approach not only enables knowledge transfer across models of different scales, training stages, or generations but also recovers traditional OPD as a special case.

3. Experiments and Results

The research team conducted systematic experiments, yielding the following findings:

1. A stronger expert is not necessarily better: Task-specific training (e.g., reinforcement learning and supervised fine-tuning) induces differences that transfer consistently, while those arising from model scale alone transfer weakly or not at all.
2. Complementary roles of expert and reference: The expert provides transferable capabilities, while the reference filters out shared behaviors and isolates relevant residuals.
3. Criteria for successful transfer: Even a useful difference does not transfer to every target; success requires the induced update to move the target towards the expert and generalize beyond sampled rollouts. Raw policy distance does not predict transferability, whereas an appropriate reference can rescue otherwise unsuccessful transfer.

4. Engineering Trade-offs and Practical Deployment Recommendations

The advantages of this method include its flexibility and wide applicability, making it suitable for models of different sizes and training stages. However, the main challenge lies in selecting appropriate proxy experts and reference policies, as well as balancing transfer efficiency and model performance. Additionally, the method demands high computational resources, especially when dealing with large-scale models, necessitating optimization of the training process to reduce computational costs.

5. Future Directions

The OPD method based on relative policy differences opens new avenues for knowledge transfer in reinforcement learning. Future research could explore its applications in multi-agent systems, continual learning, and cross-task transfer. Furthermore, combining this method with other advanced technologies, such as meta-learning and self-supervised learning, could further enhance its performance and applicability.

---

Source: Hugging Face Daily Papers (2026-10-09)

— END —

Tags: #Hugging Face Follow #Reinforcement Learning Follow #On-Policy Distillation Follow #Knowledge Transfer Follow #AI Agents Follow