Hugging Face Proposes iADD: Enhancing Alignment and Diversity in Diffusion Policy Optimization

Summary:Hugging Face's research team introduces iADD, a novel approach to improve the reinforcement learning-based post-training of diffusion models, particularly focusing on optimizing the trade-off between alignment and diversity during reward optimization. By employing theoretical analysis and an incremental Feynman-Kac training method, iADD demonstrates significant performance gains in balancing alignment and diversity across multiple tasks. Experimental results validate its effectiveness, showcasin

---

Background and Challenges

In the post-training process of reinforcement learning-based diffusion models, such as Denoising Diffusion Policy Optimization (DDPO), models optimize a reverse diffusion process under a reward function. However, current reward optimization approaches often come at the cost of diversity and quality. iADD aims to improve this process through theoretical analysis and innovative method design.

Key Innovations

1. Theoretical Analysis: iADD provides a theoretical framework that mathematically demonstrates that updates to only the latter timesteps of the diffusion model may be detrimental to diversity, contrary to conclusions presented in prior work.

2. Incremental Feynman-Kac Training: Based on strong theoretical foundations, iADD proposes an incremental Feynman-Kac training method to achieve the best alignment-diversity trade-offs.

3. Experimental Validation: iADD is extensively compared against related diffusion policy optimization approaches across three different tasks, with strong ablation studies for each component, validating its significant performance gains in both alignment and diversity.

Technical Highlights

• Theory-Driven Innovation: iADD goes beyond experimental results by employing rigorous theoretical analysis to guide its method design.
• Incremental Training Method: The incremental Feynman-Kac training allows iADD to maintain efficient training while enhancing model alignment and diversity.
• Multi-Task Applicability: The method demonstrates strong performance across multiple tasks, showcasing its wide applicability.

Industry Impact and Developer Recommendations

iADD brings new insights to the AI field, particularly in handling complex tasks and improving model performance. For developers, adopting iADD can significantly enhance AI agents' performance in complex environments. Additionally, the method opens new research directions, such as applications in multimodal tasks and further theoretical extensions.

Conclusion

Hugging Face's iADD method, through innovative theoretical analysis and training techniques, significantly improves the alignment and diversity trade-offs in the post-training process of diffusion models, offering a new technical pathway for AI agents in complex tasks.

---

Source: Hugging Face Daily Papers (2026-10-01)

— END —

Tags: #Hugging Face Follow #Diffusion Models Follow #Reinforcement Learning Follow #Alignment and Diversity Follow #Feynman-Kac Training Follow