Reinforcement Learning from Human Feedback (RLHF)
Reinforcement Learning from Human Feedback (RLHF) is an alignment methodology that trains Large Language Models (LLMs) using human preference rankings as reward signals via reinforcement learning algorithms. It transforms models from raw probabilistic text imitators (next-token predictors) into responsive, instruction-following assistants aligned with the core 3H criteria: Helpful, Honest, and Harmless.
Three-Stage Lifecycle and Mathematical Formulations
Standard RLHF pipelines follow three sequential stages:
[Stage 1: Supervised Fine-Tuning (SFT)]
Curated instruction-response dataset ──► Cross-entropy tuning ──► Baseline policy π_SFT
│
▼
[Stage 2: Reward Model (RM) Training]
Prompt x with sampled pair (y_w, y_l) ──► Human preference rankings ──► Reward model r_ψ(x, y)
│
▼
[Stage 3: PPO Reinforcement Learning]
Prompt ──► Policy network π_θ(y|x) ──► Emits tokens y ──► Reward Model r_ψ score
▲ │
│ KL Divergence Penalty ▼
└────────── PPO gradient update ◄────── Combined Reward R(x, y)
(Frozen Reference π_ref)
1. Supervised Fine-Tuning (SFT)
The base foundation model is fine-tuned on high-quality demonstration pairs via standard causal language modeling objectives, producing an initialized conversational policy $\pi^{\text{SFT}}$.
2. Reward Model (RM) Training
The reward model ($r_\psi(x, y)$) takes a prompt $x$ and a candidate completion $y$, outputting a scalar score quantifying alignment with human preference.
It is parameterized using the Bradley-Terry (BT) preference model. For a pair of completions where annotators prefer winning response $y_w$ over losing response $y_l$, the probability of preference is modeled via the logistic sigmoid:
$$P(y_w \succ y_l \mid x) = \sigma\left(r_\psi(x, y_w) - r_\psi(x, y_l)\right) = \frac{1}{1 + e^{-(r_\psi(x, y_w) - r_\psi(x, y_l))}}$$
The reward model minimizes the negative log-likelihood loss:
$$\mathcal{L}{\text{RM}}(\psi) = -\mathbb{E}{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma\left(r_\psi(x, y_w) - r_\psi(x, y_l)\right) \right]$$
3. Policy Optimization via PPO
The policy model is initialized as $\pi_\theta = \pi^{\text{SFT}}$, alongside a frozen reference replica $\pi_{\text{ref}}$. Using Proximal Policy Optimization (PPO), the system maximizes the composite objective:
$$\max_{\theta} \mathbb{E}{x \sim \mathcal{D},\ y \sim \pi\theta(y \mid x)} \left[ r_\psi(x, y) - \beta D_{\text{KL}}\left(\pi_\theta(y \mid x) \parallel \pi_{\text{ref}}(y \mid x)\right) \right] + \gamma \mathbb{E}{x \sim \mathcal{D}{\text{pretrain}}} \left[ \log \pi_\theta(x) \right]$$
Component Breakdown:
- $r_\psi(x, y)$: Scalar reward signal emitted by the learned reward model;
- $D_{\text{KL}}\left(\pi_\theta \parallel \pi_{\text{ref}}\right)$ (KL Divergence Penalty): $$D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}}) = \sum_{t=1}^T \log \frac{\pi_\theta(y_t \mid x, y_{<t})}{\pi_{\text{ref}}(y_t \mid x, y_{<t})}$$ Prevents policy collapse and mode drift. Without this constraint, the policy exploits reward model blind spots, generating ungrammatical token gibberish that scores deceptively high;
- $\beta > 0$: Regularization coefficient bounding divergence from the base policy;
- $\gamma \log \pi_\theta(x)$: Pre-training gradient mixing (PPO-ptx) mitigating degradation in core language modeling benchmarks.
Distributed Four-Model Training Architecture
During active PPO optimization, the training infrastructure concurrently orchestrates four distinct deep neural networks in GPU memory:
┌─────────────────────────────────────────────────────────┐
│ Actor-Critic Distributed Topology │
│ │
Prompt x ──────┼──► [1. Actor Policy π_θ] (Generates tokens y; active) │
│ │
├──► [2. Reference Model π_ref] (Frozen; computes KL div) │
│ │
(x, y) ────────┼──► [3. Reward Model r_ψ] (Frozen; emits scalar reward) │
│ │
(x, y) ────────┼──► [4. Critic Network V_ϕ] (Predicts state values; GAE) │
└─────────────────────────────────────────────────────────┘
- Actor Policy ($\pi_\theta$): The target language model being aligned, sampling autoregressive tokens and receiving policy gradient updates;
- Critic Network ($V_\phi$): Estimates state values for Generalized Advantage Estimation (GAE), updated via value loss;
- Reference Model ($\pi_{\text{ref}}$): Frozen snapshot providing baseline token probability distributions for KL penalties;
- Reward Model ($r_\psi$): Frozen discriminator assigning scalar rewards to completed sequences.
This multi-model footprint introduces extreme memory overhead: training 70B parameter models mandates distributed frameworks like DeepSpeed-Chat, Ray, or Megatron-LM with tensor, pipeline, and ZeRO-3 sharding.
Production Failure Modes and Alignment Traps
- Reward Hacking and Overoptimization:
Language models aggressively exploit optimization shortcuts in surrogate reward models:
- Verbosity Bias: The policy discovers that lengthy, elaborate responses receive inflated reward scores regardless of factual substance;
- Sycophancy: The model blindly validates user biases, echoing demonstrable factual errors to maximize perceived friendliness scores.
- Hyperparameter Fragility: PPO is notorious for tuning sensitivity. Imbalances among learning rates, clipping boundaries ($\epsilon$), KL penalty scales ($\beta$), or advantage parameters ($\lambda$) precipitate training collapse or catastrophic policy entropy drops.
Architectural Evolutions: Beyond Classical PPO
Given the complexity and volatility of PPO, preference optimization has bifurcated into streamlined alternatives:
| Methodology | Optimization Engine | Dedicated RM / Critic | Primary Domain |
|---|---|---|---|
| Classical RLHF (PPO) | Closed-loop online exploration via Actor-Critic | Yes (Requires 4 concurrent models) | Subjective open-ended dialogue, general conversational assistants |
| Direct Preference Optimization (DPO) | Analytically parameterizes reward via implicit log-ratios | No (Actor + Reference only) | Efficient preference alignment, stylistic tuning |
| Rule-Based Verifiable Rewards (RLVR) | Replaces neural reward models with deterministic compilers and math verifiers | No (Deterministic ground-truth verifiers) | Complex reasoning models (DeepSeek-R1, OpenAI o1), competitive programming |
Sources and Key Literature
- Training language models to follow instructions with human feedback (InstructGPT, Ouyang et al., NeurIPS 2022) — Seminal OpenAI paper establishing the modern RLHF paradigm.
- Deep Reinforcement Learning from Human Preferences (Christiano et al., NeurIPS 2017) — Foundational theory for learning reward models from trajectory comparisons.
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., NeurIPS 2023) — Closed-form preference optimization bypassing explicit RM and PPO loops.
- Secrets of RLHF in Large Language Models (Sun et al., 2023) — Empirical guidelines for stabilizing RLHF and mitigating reward hacking.