Skip to main content
ZICQ

Wiki Concepts

Model Distillation

Concepts
Aliases: knowledge distillation ·2026-09-14

Model Distillation

Model Distillation is the process of "distilling" knowledge from a large model (teacher) into a smaller model (student). The student learns from the teacher's "soft labels" (probability distributions, not hard labels), achieving near-teacher performance at an order of magnitude fewer parameters.

Classic formula

L = α · L_hard(y_true, y_student) + (1 - α) · L_soft(softmax(z_T / T), softmax(z_S / T))
  • T (temperature): softens logits, letting student learn "dark knowledge" (hidden class relations).
  • L_soft: KL divergence, student mimics teacher's output distribution.
  • L_hard: standard cross-entropy.
  • α: weight between the two losses.

Distillation in the LLM era

The goal has changed: let a small model learn the output text of a large model (instruction pairs), not the logit distribution.

Representative cases:

  • DeepSeek-R1 distillation (2025): 800k high-quality samples from R1, distilled 1.5B-70B models at all sizes, much stronger than same-size direct SFT.
  • Orca / Orca-2: distill 13B model with GPT-4 reasoning traces.
  • Alpaca / Vicuna: early classic cases.
  • TinyLlama: trained from scratch + large model data.

Distillation vs supervised fine-tuning

Dimension Traditional SFT Distillation
Data Human-written (instruction, answer) Large model generated
Cost High (manual) Low (API calls)
Data quality High Depends on teacher
Diversity Limited Large (batch-generatable)
Reasoning ability Hard to inherit Teacher's CoT inheritable

Practical advice

  • Teacher must be strong: GPT-4 / Claude / R1 class.
  • Data must be diverse: cover various task types, don't distill one prompt pattern.
  • Filter garbage: teacher also makes mistakes, filter with second model as critic.
  • Small models can distill strong capability: Llama 3.1 8B distilled approaches 70B.

Caveats

  • Distillation inherits bias: teacher's biases are inherited too.
  • Copyright issues: using ChatGPT output to train commercial models has risk (OpenAI ToS prohibits).
  • Capability ceiling: student usually can't catch up to teacher, gap grows with compression ratio.