Hugging Face Introduces RSIPO: Revolutionizing Agentic Reinforcement Learning with Skill-Policy Co-Evolution

Summary:Hugging Face has introduced Recursive Self-Improving Policy Optimization (RSIPO), a novel framework for agentic reinforcement learning (RL) that addresses credit assignment challenges in long-horizon tasks. RSIPO enables the co-evolution of agentic skills and the policy model by allowing the policy to extract skills from completed trajectories and optimize its skill-generation capability through trajectory replay. Additionally, it employs empirical Bayes shrinkage to dynamically calibrate the st

---

1. Introduction

In recent years, the credit assignment problem in long-horizon tasks has been a major focus in agentic reinforcement learning (RL). Traditional methods often struggle with effectively assigning credit in complex tasks, leading to inefficient learning. Hugging Face's Recursive Self-Improving Policy Optimization (RSIPO) framework offers a novel solution by enabling the co-evolution of skills and policies.

2. Technical Mechanism Analysis

The core mechanisms of RSIPO include:

• Skill Extraction and Policy Optimization: RSIPO allows the policy model to extract skills from completed trajectories and optimize its skill-generation capability through trajectory replay. This mechanism ensures continuous improvement of the policy model's decision-making abilities.

• Empirical Bayes Shrinkage: To address the potential unreliability of skill-enhanced teacher guidance, RSIPO employs empirical Bayes shrinkage to dynamically calibrate the student's trust in teacher scores. This approach maintains the advantages of teacher guidance while reducing reliance on unreliable guidance.

• Co-Evolution Framework: RSIPO's co-evolution framework enables the continuous improvement of the agent during training. This framework not only enhances the model's generalization capabilities but also improves its performance in complex tasks.

3. Engineering Trade-offs and Experimental Performance

The main challenges in implementing RSIPO include computational complexity and training stability. By optimizing algorithms and employing parallel computing techniques, RSIPO maintains efficient training while significantly improving model performance.

Experimental results demonstrate RSIPO's superior performance on the ALFWorld task, achieving a 98.6% success rate and outperforming the second-best baseline and standard GRPO by 2.5% and 17.4%, respectively. Additionally, RSIPO excels in search-based agentic tasks, showcasing its strong potential in long-horizon task handling.

4. Developer Implementation and Deployment Recommendations

For developers, RSIPO provides an efficient training framework that can significantly enhance the performance of agents in complex tasks. Here are some deployment recommendations:

• Resource Management: Due to the large amount of trajectory data RSIPO needs to process, it is advisable to manage computing resources effectively to avoid memory bottlenecks.

• Model Evaluation: Before deployment, it is recommended to thoroughly evaluate the RSIPO model to ensure its stability and reliability across different tasks.

• Continuous Optimization: The RSIPO framework supports continuous optimization. It is advisable to continuously collect data and improve the model during application to achieve the best performance.

5. Conclusion

The RSIPO framework, through the co-evolution of skills and policies, provides a new approach to agentic reinforcement learning. Its excellent performance in tasks like ALFWorld demonstrates the framework's great potential in long-horizon task handling. Hugging Face's innovation opens up new directions for the development of agent technology.

---

Source: Hugging Face Daily Papers (2026-10-11)

— END —

Tags: #Hugging Face Follow #Reinforcement Learning Follow #Agentic Systems Follow #RSIPO Follow #Qwen Follow