Matched-Compute Evaluation Reverses Dense-Reward Advantages in GRPO
Summary:A study of reinforcement learning with verifiable rewards evaluates dense process rewards using a matched-compute protocol across six models and 1,668 runs. Monte-Carlo continuation-based rewards appear stronger per update, but continuation sampling consumes 91% of generated tokens and reverses the ranking when token and GPU-hour budgets are matched. Per-token gradient tests overestimate the gains by one to two percentage points. Selecting prompts with predicted success probability near 50% prov
Core Finding: Compare Reward Methods Under Matched Compute
GRPO trains a policy from relative rewards across a group of sampled solutions, making it useful for math reasoning with automatically verifiable answers. Dense process rewards attempt to assign credit to intermediate reasoning steps, but methods such as Monte-Carlo continuation scoring require additional rollouts. Those rollouts can dominate the actual optimization cost.
The study's main contribution is an evaluation protocol rather than a new reward function. Every intervention receives the same generated-token budget and the same training GPU-hour budget. Without this normalization, an apparent advantage per update may simply reflect more computation spent per update.
Why Dense Rewards Can Look Stronger at the Wrong Scale
Standard GRPO samples multiple answers for the same problem, computes relative advantages from final-answer correctness, and distributes sequence-level rewards across tokens. A Monte-Carlo process reward samples further continuations from intermediate steps and estimates how often each step leads to a correct answer.
This creates substantial overhead. In the reported setting, continuation generation alone accounts for 91% of all generated tokens. A comparison based only on accuracy after one update therefore favors the dense reward simply because it performs more sampling and forward computation.
Token matching alone is also insufficient. Different reward mechanisms alter sequence lengths, sampling counts, batching behavior, and GPU utilization. The study therefore reports both equal-token and equal-GPU-time comparisons.
How 1,668 Runs Reverse the Ranking
The experiments span six models and 1,668 runs, using preregistered studies and controls for side effects. On Qwen2.5-1.5B, Monte-Carlo dense rewards outperform standard GRPO when judged per update. Yet at equal generated-token budgets, final accuracy is 14.3 percentage points lower; at equal training time, it is 7.8 points lower. None of four dense-reward variants beats GRPO even when granted additional training time.
Fine-grained credit assignment is not automatically better than terminal rewards. Its potential advantages must offset branch sampling, reward computation, and potentially longer trajectories. Ignoring those costs can turn a resource difference into a false algorithmic conclusion.
Why Per-Token Gradient Tests Mislead
The paper also evaluates a diagnostic that compares per-token gradients on a single batch. Its ranking broadly agrees with full training, but it overestimates each reward's benefit by one to two percentage points—possibly as much as the gain the test is intended to detect.
Gradient diagnostics typically omit training dynamics, sampling variance, and finite-batch noise. A step that shows stronger gradient correlation under the current policy does not guarantee persistent accuracy gains over thousands of updates. Such tests are useful for early screening, but they are not substitutes for full matched-compute training.
A Cheaper Alternative: Select Prompts Near 50% Success
The study finds that prompt selection can be more efficient than dense rewards. Using predicted success probability, training can focus on examples near 50% success: not already mastered, but not uniformly beyond the model's current capability.
Under matched compute, this strategy improves accuracy by 1.7 points on Qwen3-0.6B and 0.8 points on Qwen2.5-1.5B. The gains replicate with fresh random seeds and unseen problems. Across six preregistered tests, a simple heuristic predicted where the gains appeared in five: candidate prompts should have a predicted failure probability above 50%, equivalently a success probability below 50%.
Engineering Trade-offs and Reproduction Boundaries
The study offers a stronger protocol for RLVR experimentation:
- Account for generated-token volume and GPU time simultaneously, rather than normalizing by update count alone.
- Track rollout generation, reward estimation, forward passes, and sequence lengths as separate cost components.
- Use preregistered hypotheses and fresh seeds to reduce post-hoc selection effects.
- Test across models before attributing a result to a generic property of dense rewards.
Matched compute does not automatically guarantee external validity. Cluster hardware, batch size, sequence length, and sampling implementation can change utilization. Success-probability estimates may also depend on the model and task distribution. Whether dense rewards remain uneconomical under longer horizons, distillation settings, or inference-time supervision still requires independent validation.
Practical Recommendations for RLVR Teams
- Do not compare methods by update count. Report equal-token and equal-GPU-hour results.
- Track reward overhead separately. Include rollout branches, generated tokens, and reward-model or verifier time.
- Try prompt selection first. Prioritize examples with predicted success probabilities near the learning frontier.
- Keep GRPO as a strong baseline. Dense rewards should beat GRPO under matched budgets, not merely per update.
- Inspect learning dynamics. Determine whether gains are early-training effects and whether simpler alternatives, such as a higher initial learning rate, reproduce them.
The central takeaway is methodological: process-reward research must include reward estimation in its compute model. Otherwise, an expensive evaluator can create the appearance of algorithmic progress that is actually driven by unequal budgets.
— END —Source: Hugging Face Daily Papers (2026-11-10)
Tags: #GRPO #RLVR #Process Rewards #Matched Compute #Reinforcement Learning
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments