Skip to main content
ZICQ

Wiki Technical Terms

GRPO (Group Relative Policy Optimization)

Technical Terms
Aliases: Group Relative Policy Optimization ·2026-11-18

GRPO (Group Relative Policy Optimization)

GRPO (Group Relative Policy Optimization) is a reinforcement learning policy optimization algorithm introduced by DeepSeek in the 2024 DeepSeekMath paper. Building on PPO's clipped surrogate objective, GRPO eliminates the separate value (critic) network and instead estimates advantage baselines directly from a group of sampled completions for the same prompt. This roughly halves RL training memory overhead and has become a workhorse optimizer for RLVR and reasoning models.

Why Standard PPO Struggles on Reasoning Tasks

Conventional RLHF / PPO requires co-locating four large models in GPU memory—policy (actor), reference, reward, and critic. For long-horizon reasoning:

  • Memory Pressure: The critic typically matches the actor's parameter count, doubling optimizer and activation footprints.
  • Sparse Outcome Rewards: Math proofs and unit tests yield sparse final binary rewards; training a stable critic to score arbitrary intermediate tokens across thousands of reasoning tokens is brittle.

Core Mechanism: Group-Relative Baselines

GRPO replaces the learned value function with empirical within-group normalization:

  1. Group Rollout: For each prompt q, sample a group of G full completions {o1,o2,…,oG} from πθold.
  2. Scoring: Evaluate each completion with a deterministic verifier (such as math equivalence checking or test suites) or a reward model to obtain scores {R1,R2,…,RG}.
  3. Relative Advantage: Normalize each score against the group mean and standard deviation:
Ai=Ri−mean(R1,…,RG)std(R1,…,RG)
  1. Clipped Update + KL Regularization: Update θ using a PPO-style clipped ratio objective while incorporating KL divergence against πref directly in the loss term rather than inside per-token reward shaping.

Comparison with PPO and DPO

Dimension PPO DPO GRPO
Requires Critic Network Yes (high VRAM cost) No (offline) No (group statistics baseline)
Data Regime Online rollouts Offline pairs (yw,yl) Online group rollouts
Primary Fit General preference & token-level shaping Style, tone, and safety alignment Verifiable math, code, and reasoning (RLVR)
Main Bottleneck Multi-model orchestration Length bias & off-policy drift Generation throughput during rollouts

Engineering Considerations

  • Rollout-Bound Training: With the critic removed, sampling G long trajectories per prompt dominates runtime; RL frameworks integrate vLLM or SGLang for fast batched generation.
  • Zero-Variance Groups & Length Bias: Prompts where all G samples pass or fail yield std(R)=0 and zero gradient; pipelines commonly filter trivial/impossible prompts or apply unnormalized group-mean variants (e.g., Dr. GRPO) to curb response-length inflation.
  • Verifier Integrity: Enforce structural tags (e.g., <think>...</think>) alongside answer correctness checks so models cannot game parser quirks.

Further Reading