ZICQ
中 Log in / Sign up
Newsroom LLMs & Foundation Models #Hugging Face #EasyPPO #Reinforcement Learning #PPO Algorithm #Large Language Models

Hugging Face Releases EasyPPO: Addressing Critic Instability in PPO Training

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 8 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:Hugging Face has introduced EasyPPO, a novel approach designed to address the instability issues of the critic in Proximal Policy Optimization (PPO) for reinforcement learning. EasyPPO employs actor-only overlong filtering, noise-normalized critic regression, and moderately smaller critic mini-batches to tackle problems arising from truncated trajectory processing and heterogeneous return noise. These innovations shift the policy objective to include both completed and truncated returns and bala


Background and Challenges

In reinforcement learning, Proximal Policy Optimization (PPO) is favored for its learned critic, which leverages historical trajectories to estimate expected returns and reduce policy gradient variance. However, the instability of the critic remains a significant challenge, particularly when dealing with Large Language Models (LLMs). The critic's instability primarily stems from two key issues:

  1. Truncated Trajectory Processing Issue: Filtering truncated trajectories from both the actor and critic shifts the policy objective towards rewards conditioned on completion, allowing truncation to increase even as conditional returns improve.
  2. Heterogeneous Return Noise Issue: In finite batches, high-variance prompts can dominate critic updates, leading to training instability.

EasyPPO's Solution

To address these challenges, Hugging Face has introduced EasyPPO, featuring the following core innovations:

  • Actor-Only Overlong Filtering: By training the critic on returns from both completed and truncated trajectories, it prevents the policy objective from shifting.
  • Noise-Normalized Critic Regression: Each prompt's critic loss is weighted by the inverse standard deviation of its sampled returns, balancing noise contributions across prompts.
  • Moderately Smaller Critic Mini-Batches: This confines the influence of outliers to fewer trajectories during gradient clipping.

Experimental Results

EasyPPO was tested across multiple benchmarks, including continuous-reward coding on FrontierCS, binary-reward mathematical reasoning on AIME24, and multi-turn search on Search-R1. The results demonstrate that EasyPPO remains stable throughout the training horizon and outperforms traditional PPO, VAPO, and HL-Gauss PPO by 14.89%, 2.28%, and 9.47% in validation scores, respectively.

Industry Impact and Developer Recommendations

The release of EasyPPO marks a significant advancement in reinforcement learning, particularly in handling complex tasks and large language models. Here are some recommendations:

  • Developers: Consider adopting EasyPPO in PPO training to enhance stability and performance.
  • Researchers: Explore the potential of EasyPPO in different application scenarios and optimize it further with other techniques.
  • Enterprise Users: In applications requiring efficient reinforcement learning, EasyPPO can be a valuable tool to improve efficiency.

Source: Hugging Face Daily Papers (2026-09-29)

— END —

Tags: #Hugging Face #EasyPPO #Reinforcement Learning #PPO Algorithm #Large Language Models

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…