In-Context Learning (ICL)
In-context learning (ICL) is one of the most important capabilities of LLMs: putting a few "input-output examples" or "instructions" in the prompt lets the model perform a task without updating weights. It blurs the line between "training" and "inference" — the same model can do countless tasks just by swapping the prompt.
Three typical forms
- Zero-shot: instruction only, no examples. "Translate the following Chinese to English: …"
- One-shot: 1 example + task
- Few-shot: 3-10 examples + task (the classic paradigm from the GPT-3 paper)
Why it matters
- No fine-tuning: traditional deep learning requires retraining per task; ICL turns "swap prompt" into "swap capability" at near-zero cost
- Instant: new requirements work immediately; cross-language / cross-task switching is free
- Composable: multiple examples can demonstrate complex reasoning / tool calls / agent behavior
- Interpretable: humans see every example in the prompt and know what the model is "thinking"
- Personalization: per-user / per-scene prompt adjustments don't affect the underlying model
How it works (rough theories)
Three complementary explanations for ICL's success:
- Implicit Bayesian inference: the model infers latent concepts from examples, similar to learning the conditional distribution of the task
- Task location: the model contains many latent task vectors in its activations; few-shot examples "surface" the right one
- Meta-learning: trained on a vast task distribution; inference is "just picking a task in distribution" — this is the central finding of Anthropic / DeepMind transformer-circuits work
Advanced uses
- Chain-of-Thought (CoT): examples include reasoning steps; model learns to write out intermediate steps
- Self-consistency: sample multiple times and majority-vote
- Tool-use prompting: examples include tool calls; model learns to invoke structured tools
- Retrieval-augmented ICL: dynamically pull examples from an external database
- Many-shot ICL: Claude 2.1 / Gemini 1.5 use 1000+ token example blocks to lift long-tail tasks (the "many-shot is still zero-shot" finding)
Boundaries and failure modes
- Sensitive to example order and noise: permuting or corrupting examples can swing results
- Complex reasoning still needs CoT: pure ICL fails on multi-step math / logic
- Long-prompt cost: more tokens = more cost, but the benefit isn't linear
- Can't learn new capabilities: only "surfaces" latent tasks the model already has