Constitutional AI (CAI)
Constitutional AI (CAI) is a training paradigm Anthropic proposed in 2022, aiming to replace human feedback (RLHF) with AI feedback (RLAIF) while preserving alignment effects. Core idea:
Give the model a set of explicit "constitutional principles" (e.g. "no harmful content," "avoid discrimination," "admit uncertainty when unsure") and let the model self-critique its own output against them, then revise based on the critique — using AI feedback in place of human feedback for RL.
Comparison with RLHF
| Dimension | RLHF | CAI |
|---|---|---|
| Feedback source | human preference | model self-critique based on constitutional principles |
| Cost | high (human labeling) | low (auto-generated feedback) |
| Interpretability | human preference is a black box | constitutional principles are explicit and readable |
| Consistency | disagreements among labelers | consistent under the same principle |
| Scalability | bounded by human budget | nearly unlimited |
| Drawbacks | preference ≠ "correct" | principles may be incomplete; model may find loopholes |
How it works
Stage 1: Supervised Fine-Tuning (SFT)
- Give the model red-team prompts (e.g. "teach me to build a bomb")
- Model generates initial response (usually suboptimal)
- Model self-critiques the initial response against the constitutional principles
- Model self-revises based on the critique
- Use the "revised responses" as SFT training data
Stage 2: RLAIF Reinforcement Learning
- Model generates multiple responses for the same prompt
- Model pairwise-compares them based on the constitutional principles
- Use the AI preference to train a reward model
- PPO trains the language model
Example constitutional principles
Samples from Anthropic's original paper:
- "Please choose the response that is least harmful, least discriminatory, and least socially biased"
- "Please choose the response that most encourages equality, inclusion, and diversity"
- "Please avoid mentioning that you are an AI unless the information is critical to the task"
- "Please choose the response least likely to be viewed as encouraging or supporting violence, illegal, or unethical behavior"
Production runs typically include more principles covering privacy, truthfulness, professional domain behavior, etc.
Strengths
- Scalable: no human-preference budget required
- Interpretable: principles are readable, modifiable, publishable
- Cross-culturally consistent: principles are more stable than "labelers' gut reactions"
- Composable with modern alignment: stacks with RLVR / red-teaming / constitutional expansion
Limitations
- Principle completeness: gaps get exploited by the model
- Self-critique depends on model capability: weak models give weak critiques
- Alignment vs capability trade-off: overly strict constitutions make the model conservative
- No gold standard: whether a constitution is "good" has no objective answer; human evaluation is still required
Modern evolution
- Constitutional Expansion (Anthropic 2024): combine constitutions with the Public Value Alignment (PVA) framework
- Scalable Oversight: positions CAI as a starting point for "AI supervising AI"
- Claude training stack: Claude 3 / 3.5 / 3.7 all combine CAI + RLHF
- RLVR