RLVR (Reinforcement Learning with Verifiable Rewards)
RLVR is the new RL paradigm for large models that emerged from 2024 onward: the reward signal does not require human judgment but comes from a machine-verifiable program. Examples:
- Math problem correctness (automatic grader)
- Code passing unit tests
- Board / game outcomes (environment reports)
- Formal proofs checked by a theorem prover
This route sidesteps the two fundamental pain points of RLHF — human preference is both expensive and can misalign with "actually correct" — and lets the model grind on "falsifiable" tasks by itself.
Comparison with RLHF
| Dimension | RLHF | RLVR |
|---|---|---|
| Reward source | human preference rankings | program verification (right/wrong, tests pass, formal proof) |
| Cost | high (human labeling) | very low (machine verification) |
| Scalability | bounded by human budget | bounded only by compute and the scope of verifiable tasks |
| Calibration | usually overconfident (RLHF rewards confident answers) | usually better calibrated (signal is real correctness) |
| Applicable tasks | subjective / open-ended (chat, copy) | objective / verifiable (math, code, proof) |
| Failure modes | reward hacking | narrow task scope, fails out of distribution |
Key milestones
- 2022-2023: DeepMind's AlphaProof / AlphaGeometry early thinking
- 2024-09: OpenAI o1 brings RLVR to industrial scale (big jump on AIME / Codeforces)
- 2024-11: DeepSeek R1 open-sources RLVR + GRPO, triggering the whole industry to follow
- 2025: every reasoning model (Qwen QwQ, Claude 3.7 Sonnet thinking mode, Gemini 2.0 thinking) deeply adopts RLVR training
Workflow
- SFT base: supervised fine-tune so the model can already attempt the task
- Verifiable task set: collect / generate tasks whose correctness is machine-checkable
- PPO / GRPO training: model produces an answer, verifier emits a 0/1 reward, backprop updates the policy
- Reward shaping: optionally pair with a Process Reward Model (PRM) to score intermediate steps
Strengths
- No human preference needed, scales infinitely
- Significantly better calibration than RLHF
- Avoids reward hacking: reward source is real correctness, not "looks correct"
- Interpretable reasoning paths: each step is checkable, so models tend to write traceable reasoning
Limitations
- Only verifiable tasks: can't help with chat / copy / creative work
- The verifier can be wrong: a buggy grader teaches the model wrong lessons
- Over-optimization risk: models can exploit verifier loopholes (e.g. matching an answer format the grader misreads as correct)
- Out-of-distribution failure: RLVR only sharpens ability within the trained task family