Skip to main content
ZICQ

Wiki AI Concepts

RLVR (Reinforcement Learning with Verifiable Rewards)

AI Concepts
Aliases: RLVR Reinforcement Learning with Verifiable Rewards Verifiable Rewards ·2026-09-19

RLVR (Reinforcement Learning with Verifiable Rewards)

RLVR is the new RL paradigm for large models that emerged from 2024 onward: the reward signal does not require human judgment but comes from a machine-verifiable program. Examples:

  • Math problem correctness (automatic grader)
  • Code passing unit tests
  • Board / game outcomes (environment reports)
  • Formal proofs checked by a theorem prover

This route sidesteps the two fundamental pain points of RLHF — human preference is both expensive and can misalign with "actually correct" — and lets the model grind on "falsifiable" tasks by itself.

Comparison with RLHF

Dimension RLHF RLVR
Reward source human preference rankings program verification (right/wrong, tests pass, formal proof)
Cost high (human labeling) very low (machine verification)
Scalability bounded by human budget bounded only by compute and the scope of verifiable tasks
Calibration usually overconfident (RLHF rewards confident answers) usually better calibrated (signal is real correctness)
Applicable tasks subjective / open-ended (chat, copy) objective / verifiable (math, code, proof)
Failure modes reward hacking narrow task scope, fails out of distribution

Key milestones

  • 2022-2023: DeepMind's AlphaProof / AlphaGeometry early thinking
  • 2024-09: OpenAI o1 brings RLVR to industrial scale (big jump on AIME / Codeforces)
  • 2024-11: DeepSeek R1 open-sources RLVR + GRPO, triggering the whole industry to follow
  • 2025: every reasoning model (Qwen QwQ, Claude 3.7 Sonnet thinking mode, Gemini 2.0 thinking) deeply adopts RLVR training

Workflow

  1. SFT base: supervised fine-tune so the model can already attempt the task
  2. Verifiable task set: collect / generate tasks whose correctness is machine-checkable
  3. PPO / GRPO training: model produces an answer, verifier emits a 0/1 reward, backprop updates the policy
  4. Reward shaping: optionally pair with a Process Reward Model (PRM) to score intermediate steps

Strengths

  • No human preference needed, scales infinitely
  • Significantly better calibration than RLHF
  • Avoids reward hacking: reward source is real correctness, not "looks correct"
  • Interpretable reasoning paths: each step is checkable, so models tend to write traceable reasoning

Limitations

  • Only verifiable tasks: can't help with chat / copy / creative work
  • The verifier can be wrong: a buggy grader teaches the model wrong lessons
  • Over-optimization risk: models can exploit verifier loopholes (e.g. matching an answer format the grader misreads as correct)
  • Out-of-distribution failure: RLVR only sharpens ability within the trained task family