PPO (Proximal Policy Optimization), introduced by Schulman et al. at OpenAI in 2017, is a policy-gradient reinforcement learning algorithm that keeps training updates within a safe neighborhood of the previous policy. By clipping the probability ratio (or penalizing KL divergence), PPO avoids destructive policy collapses while using only first-order optimization.
During the first wave of RLHF (such as InstructGPT and early ChatGPT), PPO served as the default online optimizer. In recent post-training stacks, teams often pair offline alignment with DPO or switch online reasoning runs to GRPO to eliminate the memory footprint of a separate Critic model.
Why "Proximal" Updates Matter
Vanilla policy gradients can easily overshoot when a single large batch pushes the model far away from the rollout policy πθold, destabilizing future gradient estimates. Trust Region Policy Optimization (TRPO) solved this with costly second-order KL constraints, whereas PPO-Clip replaces them with a simple clipped surrogate objective:
Here rt(θ)=πθ(at∣st)πθold(at∣st) is the importance-sampling probability ratio, Ât is the Generalized Advantage Estimate (GAE) computed using a learned value function (Critic), and ε (commonly 0.1–0.2) caps how far an update can reward or penalize a step.
The Four-Model Footprint in LLM RLHF
Applying standard PPO to LLM token generation typically requires orchestrating four models during training:
- Actor (πθ): The policy LLM being updated, which generates full rollout responses;
- Critic (Vφ): A value network (often comparable in size to the Actor) estimating token-level expected returns to compute Ât;
- Reward Model (R): A frozen preference scorer judging the finished response;
- Reference Policy (πref): A frozen SFT snapshot supplying a per-token KL penalty −β DKL(πθ∥πref) to prevent reward hacking.
Engineering Trade-offs: PPO vs. DPO vs. GRPO
| Dimension | PPO | DPO | GRPO |
|---|---|---|---|
| Training Mode | On-policy RL | Offline pairwise preference optimization | On-policy group-relative RL |
| Requires Critic? | Yes (heavy GPU memory & tuning) | No | No (uses group-normalized baselines) |
| Signal Source | Learned reward model or verifier | Static Chosen/Rejected pairs | Verifiable rewards (RLVR) or reward models |
| Practical Role | Strong for dense token-level credit assignment and multi-turn environments | Simple, fast baseline for preference and tone alignment | Widely adopted for math/code reasoning to save Critic VRAM |
Further Reading
- Schulman et al. (2017), Proximal Policy Optimization Algorithms: https://arxiv.org/abs/1707.06347
- Ouyang et al. (2022), Training language models to follow instructions with human feedback (InstructGPT): https://arxiv.org/abs/2203.02155