Skip to main content
ZICQ

Wiki Concepts

Reinforcement Learning from Human Feedback (RLHF)

Concepts
Aliases: RLHF Reinforcement Learning from Human Feedback human feedback alignment ·2026-10-10

Reinforcement Learning from Human Feedback (RLHF)

Reinforcement Learning from Human Feedback (RLHF) is an alignment methodology that trains Large Language Models (LLMs) using human preference rankings as reward signals via reinforcement learning algorithms. It transforms models from raw probabilistic text imitators (next-token predictors) into responsive, instruction-following assistants aligned with the core 3H criteria: Helpful, Honest, and Harmless.


Three-Stage Lifecycle and Mathematical Formulations

Standard RLHF pipelines follow three sequential stages:

[Stage 1: Supervised Fine-Tuning (SFT)]
Curated instruction-response dataset ──► Cross-entropy tuning ──► Baseline policy π_SFT
                                                                      │
                                                                      ▼
[Stage 2: Reward Model (RM) Training]
Prompt x with sampled pair (y_w, y_l) ──► Human preference rankings ──► Reward model r_ψ(x, y)
                                                                      │
                                                                      ▼
[Stage 3: PPO Reinforcement Learning]
Prompt ──► Policy network π_θ(y|x) ──► Emits tokens y ──► Reward Model r_ψ score
                ▲                                              │
                │                KL Divergence Penalty         ▼
                └────────── PPO gradient update ◄────── Combined Reward R(x, y)
                            (Frozen Reference π_ref)

1. Supervised Fine-Tuning (SFT)

The base foundation model is fine-tuned on high-quality demonstration pairs via standard causal language modeling objectives, producing an initialized conversational policy $\pi^{\text{SFT}}$.

2. Reward Model (RM) Training

The reward model ($r_\psi(x, y)$) takes a prompt $x$ and a candidate completion $y$, outputting a scalar score quantifying alignment with human preference.

It is parameterized using the Bradley-Terry (BT) preference model. For a pair of completions where annotators prefer winning response $y_w$ over losing response $y_l$, the probability of preference is modeled via the logistic sigmoid:

$$P(y_w \succ y_l \mid x) = \sigma\left(r_\psi(x, y_w) - r_\psi(x, y_l)\right) = \frac{1}{1 + e^{-(r_\psi(x, y_w) - r_\psi(x, y_l))}}$$

The reward model minimizes the negative log-likelihood loss:

$$\mathcal{L}{\text{RM}}(\psi) = -\mathbb{E}{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma\left(r_\psi(x, y_w) - r_\psi(x, y_l)\right) \right]$$

3. Policy Optimization via PPO

The policy model is initialized as $\pi_\theta = \pi^{\text{SFT}}$, alongside a frozen reference replica $\pi_{\text{ref}}$. Using Proximal Policy Optimization (PPO), the system maximizes the composite objective:

$$\max_{\theta} \mathbb{E}{x \sim \mathcal{D},\ y \sim \pi\theta(y \mid x)} \left[ r_\psi(x, y) - \beta D_{\text{KL}}\left(\pi_\theta(y \mid x) \parallel \pi_{\text{ref}}(y \mid x)\right) \right] + \gamma \mathbb{E}{x \sim \mathcal{D}{\text{pretrain}}} \left[ \log \pi_\theta(x) \right]$$

Component Breakdown:

  • $r_\psi(x, y)$: Scalar reward signal emitted by the learned reward model;
  • $D_{\text{KL}}\left(\pi_\theta \parallel \pi_{\text{ref}}\right)$ (KL Divergence Penalty): $$D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}}) = \sum_{t=1}^T \log \frac{\pi_\theta(y_t \mid x, y_{<t})}{\pi_{\text{ref}}(y_t \mid x, y_{<t})}$$ Prevents policy collapse and mode drift. Without this constraint, the policy exploits reward model blind spots, generating ungrammatical token gibberish that scores deceptively high;
  • $\beta > 0$: Regularization coefficient bounding divergence from the base policy;
  • $\gamma \log \pi_\theta(x)$: Pre-training gradient mixing (PPO-ptx) mitigating degradation in core language modeling benchmarks.

Distributed Four-Model Training Architecture

During active PPO optimization, the training infrastructure concurrently orchestrates four distinct deep neural networks in GPU memory:

                ┌─────────────────────────────────────────────────────────┐
                │             Actor-Critic Distributed Topology           │
                │                                                         │
 Prompt x ──────┼──► [1. Actor Policy π_θ] (Generates tokens y; active)   │
                │                                                         │
                ├──► [2. Reference Model π_ref] (Frozen; computes KL div) │
                │                                                         │
 (x, y) ────────┼──► [3. Reward Model r_ψ] (Frozen; emits scalar reward)  │
                │                                                         │
 (x, y) ────────┼──► [4. Critic Network V_ϕ] (Predicts state values; GAE) │
                └─────────────────────────────────────────────────────────┘
  1. Actor Policy ($\pi_\theta$): The target language model being aligned, sampling autoregressive tokens and receiving policy gradient updates;
  2. Critic Network ($V_\phi$): Estimates state values for Generalized Advantage Estimation (GAE), updated via value loss;
  3. Reference Model ($\pi_{\text{ref}}$): Frozen snapshot providing baseline token probability distributions for KL penalties;
  4. Reward Model ($r_\psi$): Frozen discriminator assigning scalar rewards to completed sequences.

This multi-model footprint introduces extreme memory overhead: training 70B parameter models mandates distributed frameworks like DeepSpeed-Chat, Ray, or Megatron-LM with tensor, pipeline, and ZeRO-3 sharding.


Production Failure Modes and Alignment Traps

  1. Reward Hacking and Overoptimization: Language models aggressively exploit optimization shortcuts in surrogate reward models:
    • Verbosity Bias: The policy discovers that lengthy, elaborate responses receive inflated reward scores regardless of factual substance;
    • Sycophancy: The model blindly validates user biases, echoing demonstrable factual errors to maximize perceived friendliness scores.
  2. Hyperparameter Fragility: PPO is notorious for tuning sensitivity. Imbalances among learning rates, clipping boundaries ($\epsilon$), KL penalty scales ($\beta$), or advantage parameters ($\lambda$) precipitate training collapse or catastrophic policy entropy drops.

Architectural Evolutions: Beyond Classical PPO

Given the complexity and volatility of PPO, preference optimization has bifurcated into streamlined alternatives:

Methodology Optimization Engine Dedicated RM / Critic Primary Domain
Classical RLHF (PPO) Closed-loop online exploration via Actor-Critic Yes (Requires 4 concurrent models) Subjective open-ended dialogue, general conversational assistants
Direct Preference Optimization (DPO) Analytically parameterizes reward via implicit log-ratios No (Actor + Reference only) Efficient preference alignment, stylistic tuning
Rule-Based Verifiable Rewards (RLVR) Replaces neural reward models with deterministic compilers and math verifiers No (Deterministic ground-truth verifiers) Complex reasoning models (DeepSeek-R1, OpenAI o1), competitive programming

Sources and Key Literature