Skip to main content
ZICQ

Wiki Technical Terms

PPO (Proximal Policy Optimization)

Technical Terms
Aliases: Proximal Policy Optimization PPO-Clip ·2026-11-18

PPO (Proximal Policy Optimization), introduced by Schulman et al. at OpenAI in 2017, is a policy-gradient reinforcement learning algorithm that keeps training updates within a safe neighborhood of the previous policy. By clipping the probability ratio (or penalizing KL divergence), PPO avoids destructive policy collapses while using only first-order optimization.

During the first wave of RLHF (such as InstructGPT and early ChatGPT), PPO served as the default online optimizer. In recent post-training stacks, teams often pair offline alignment with DPO or switch online reasoning runs to GRPO to eliminate the memory footprint of a separate Critic model.

Why "Proximal" Updates Matter

Vanilla policy gradients can easily overshoot when a single large batch pushes the model far away from the rollout policy πθold, destabilizing future gradient estimates. Trust Region Policy Optimization (TRPO) solved this with costly second-order KL constraints, whereas PPO-Clip replaces them with a simple clipped surrogate objective:

LCLIP(θ)=𝔼t[min(rt(θ)Ât, clip(rt(θ), 1−ε, 1+ε)Ât)]

Here rt(θ)=πθ(at∣st)πθold(at∣st) is the importance-sampling probability ratio, Ât is the Generalized Advantage Estimate (GAE) computed using a learned value function (Critic), and ε (commonly 0.1–0.2) caps how far an update can reward or penalize a step.

The Four-Model Footprint in LLM RLHF

Applying standard PPO to LLM token generation typically requires orchestrating four models during training:

  1. Actor (πθ): The policy LLM being updated, which generates full rollout responses;
  2. Critic (Vφ): A value network (often comparable in size to the Actor) estimating token-level expected returns to compute Ât;
  3. Reward Model (R): A frozen preference scorer judging the finished response;
  4. Reference Policy (πref): A frozen SFT snapshot supplying a per-token KL penalty −β DKL(πθ∥πref) to prevent reward hacking.

Engineering Trade-offs: PPO vs. DPO vs. GRPO

Dimension PPO DPO GRPO
Training Mode On-policy RL Offline pairwise preference optimization On-policy group-relative RL
Requires Critic? Yes (heavy GPU memory & tuning) No No (uses group-normalized baselines)
Signal Source Learned reward model or verifier Static Chosen/Rejected pairs Verifiable rewards (RLVR) or reward models
Practical Role Strong for dense token-level credit assignment and multi-turn environments Simple, fast baseline for preference and tone alignment Widely adopted for math/code reasoning to save Critic VRAM

Further Reading