Model Distillation
Model Distillation is the process of "distilling" knowledge from a large model (teacher) into a smaller model (student). The student learns from the teacher's "soft labels" (probability distributions, not hard labels), achieving near-teacher performance at an order of magnitude fewer parameters.
Classic formula
L = α · L_hard(y_true, y_student) + (1 - α) · L_soft(softmax(z_T / T), softmax(z_S / T))
- T (temperature): softens logits, letting student learn "dark knowledge" (hidden class relations).
- L_soft: KL divergence, student mimics teacher's output distribution.
- L_hard: standard cross-entropy.
- α: weight between the two losses.
Distillation in the LLM era
The goal has changed: let a small model learn the output text of a large model (instruction pairs), not the logit distribution.
Representative cases:
- DeepSeek-R1 distillation (2025): 800k high-quality samples from R1, distilled 1.5B-70B models at all sizes, much stronger than same-size direct SFT.
- Orca / Orca-2: distill 13B model with GPT-4 reasoning traces.
- Alpaca / Vicuna: early classic cases.
- TinyLlama: trained from scratch + large model data.
Distillation vs supervised fine-tuning
| Dimension | Traditional SFT | Distillation |
|---|---|---|
| Data | Human-written (instruction, answer) | Large model generated |
| Cost | High (manual) | Low (API calls) |
| Data quality | High | Depends on teacher |
| Diversity | Limited | Large (batch-generatable) |
| Reasoning ability | Hard to inherit | Teacher's CoT inheritable |
Practical advice
- Teacher must be strong: GPT-4 / Claude / R1 class.
- Data must be diverse: cover various task types, don't distill one prompt pattern.
- Filter garbage: teacher also makes mistakes, filter with second model as critic.
- Small models can distill strong capability: Llama 3.1 8B distilled approaches 70B.
Caveats
- Distillation inherits bias: teacher's biases are inherited too.
- Copyright issues: using ChatGPT output to train commercial models has risk (OpenAI ToS prohibits).
- Capability ceiling: student usually can't catch up to teacher, gap grows with compression ratio.