GRPO (Group Relative Policy Optimization)
GRPO (Group Relative Policy Optimization) is a reinforcement learning policy optimization algorithm introduced by DeepSeek in the 2024 DeepSeekMath paper. Building on PPO's clipped surrogate objective, GRPO eliminates the separate value (critic) network and instead estimates advantage baselines directly from a group of sampled completions for the same prompt. This roughly halves RL training memory overhead and has become a workhorse optimizer for RLVR and reasoning models.
Why Standard PPO Struggles on Reasoning Tasks
Conventional RLHF / PPO requires co-locating four large models in GPU memory—policy (actor), reference, reward, and critic. For long-horizon reasoning:
- Memory Pressure: The critic typically matches the actor's parameter count, doubling optimizer and activation footprints.
- Sparse Outcome Rewards: Math proofs and unit tests yield sparse final binary rewards; training a stable critic to score arbitrary intermediate tokens across thousands of reasoning tokens is brittle.
Core Mechanism: Group-Relative Baselines
GRPO replaces the learned value function with empirical within-group normalization:
- Group Rollout: For each prompt q, sample a group of G full completions {o1,o2,…,oG} from πθold.
- Scoring: Evaluate each completion with a deterministic verifier (such as math equivalence checking or test suites) or a reward model to obtain scores {R1,R2,…,RG}.
- Relative Advantage: Normalize each score against the group mean and standard deviation:
Ai=Ri−mean(R1,…,RG)std(R1,…,RG)
- Clipped Update + KL Regularization: Update θ using a PPO-style clipped ratio objective while incorporating KL divergence against πref directly in the loss term rather than inside per-token reward shaping.
Comparison with PPO and DPO
| Dimension | PPO | DPO | GRPO |
|---|---|---|---|
| Requires Critic Network | Yes (high VRAM cost) | No (offline) | No (group statistics baseline) |
| Data Regime | Online rollouts | Offline pairs (yw,yl) | Online group rollouts |
| Primary Fit | General preference & token-level shaping | Style, tone, and safety alignment | Verifiable math, code, and reasoning (RLVR) |
| Main Bottleneck | Multi-model orchestration | Length bias & off-policy drift | Generation throughput during rollouts |
Engineering Considerations
- Rollout-Bound Training: With the critic removed, sampling G long trajectories per prompt dominates runtime; RL frameworks integrate vLLM or SGLang for fast batched generation.
- Zero-Variance Groups & Length Bias: Prompts where all G samples pass or fail yield std(R)=0 and zero gradient; pipelines commonly filter trivial/impossible prompts or apply unnormalized group-mean variants (e.g., Dr. GRPO) to curb response-length inflation.
- Verifier Integrity: Enforce structural tags (e.g.,
<think>...</think>) alongside answer correctness checks so models cannot game parser quirks.
Further Reading
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (Shao et al., 2024) — Original GRPO formulation
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (DeepSeek-AI, 2025) — Large-scale GRPO application in open reasoning models