Skip to main content
ZICQ

Wiki AI Concepts

Constitutional AI (CAI)

AI Concepts
Aliases: Constitutional AI CAI RLAIF ·2026-09-19

Constitutional AI (CAI)

Constitutional AI (CAI) is a training paradigm Anthropic proposed in 2022, aiming to replace human feedback (RLHF) with AI feedback (RLAIF) while preserving alignment effects. Core idea:

Give the model a set of explicit "constitutional principles" (e.g. "no harmful content," "avoid discrimination," "admit uncertainty when unsure") and let the model self-critique its own output against them, then revise based on the critique — using AI feedback in place of human feedback for RL.

Comparison with RLHF

Dimension RLHF CAI
Feedback source human preference model self-critique based on constitutional principles
Cost high (human labeling) low (auto-generated feedback)
Interpretability human preference is a black box constitutional principles are explicit and readable
Consistency disagreements among labelers consistent under the same principle
Scalability bounded by human budget nearly unlimited
Drawbacks preference ≠ "correct" principles may be incomplete; model may find loopholes

How it works

Stage 1: Supervised Fine-Tuning (SFT)

  1. Give the model red-team prompts (e.g. "teach me to build a bomb")
  2. Model generates initial response (usually suboptimal)
  3. Model self-critiques the initial response against the constitutional principles
  4. Model self-revises based on the critique
  5. Use the "revised responses" as SFT training data

Stage 2: RLAIF Reinforcement Learning

  1. Model generates multiple responses for the same prompt
  2. Model pairwise-compares them based on the constitutional principles
  3. Use the AI preference to train a reward model
  4. PPO trains the language model

Example constitutional principles

Samples from Anthropic's original paper:

  • "Please choose the response that is least harmful, least discriminatory, and least socially biased"
  • "Please choose the response that most encourages equality, inclusion, and diversity"
  • "Please avoid mentioning that you are an AI unless the information is critical to the task"
  • "Please choose the response least likely to be viewed as encouraging or supporting violence, illegal, or unethical behavior"

Production runs typically include more principles covering privacy, truthfulness, professional domain behavior, etc.

Strengths

  • Scalable: no human-preference budget required
  • Interpretable: principles are readable, modifiable, publishable
  • Cross-culturally consistent: principles are more stable than "labelers' gut reactions"
  • Composable with modern alignment: stacks with RLVR / red-teaming / constitutional expansion

Limitations

  • Principle completeness: gaps get exploited by the model
  • Self-critique depends on model capability: weak models give weak critiques
  • Alignment vs capability trade-off: overly strict constitutions make the model conservative
  • No gold standard: whether a constitution is "good" has no objective answer; human evaluation is still required

Modern evolution

  • Constitutional Expansion (Anthropic 2024): combine constitutions with the Public Value Alignment (PVA) framework
  • Scalable Oversight: positions CAI as a starting point for "AI supervising AI"
  • Claude training stack: Claude 3 / 3.5 / 3.7 all combine CAI + RLHF
    • RLVR