Skip to main content
ZICQ

Wiki AI Concepts

Diffusion Model

AI Concepts
Aliases: LDM Latent Diffusion DiT Rectified Flow ·2026-11-18

Diffusion Model

A diffusion model is a class of generative models that learn to synthesize data by iteratively removing noise from a random Gaussian sample. Diffusion and its flow-matching successors power modern high-resolution image, video, and audio generation systems—offering markedly better training stability and mode coverage than early Generative Adversarial Networks (GANs).

Core Mechanism: Forward Noising and Reverse Denoising

Diffusion operates through two coupled processes:

  1. Forward Diffusion Process: Starting from a clean training sample x0, Gaussian noise is incrementally added over T steps according to a variance schedule until the sample converges to isotropic noise xT∼𝒩(0,I).
  2. Reverse Denoising Process: A neural network εθ(xt,t,c) learns to predict the noise (or velocity field) at timestep t conditioned on prompt or control signal c. At inference time, the model starts from pure noise xT and iteratively denoises back to a coherent output x0.
Forward Noising (fixed schedule for training):
Clean sample x_0 ──► noisy x_1 ──► ... ──► Gaussian noise x_T

Reverse Denoising (learned sampling at inference):
Pure noise x_T ──► predict & subtract noise ──► ... ──► conditioned output x_0

Architectural Evolution: DDPM to LDM, DiT, and Rectified Flow

Generation Representative Work Architectural Shift Engineering Impact
Pixel-Space Diffusion DDPM (2020) Showed variational denoising matches or beats GAN fidelity High compute and memory cost from operating directly on pixel grids
Latent Diffusion (LDM) Stable Diffusion 1.x/2.x / SDXL Compresses images via a VAE into a compact latent space before denoising Cuts VRAM and FLOPs dramatically, enabling consumer-GPU inference
Diffusion Transformers (DiT) DiT / Sora / PixArt Replaces convolutional U-Nets with Vision Transformer blocks Exhibits predictable compute scaling laws and generalizes naturally to video patches
Flow Matching / Rectified Flow SD3 / Flux Learns straight-line ODE trajectories between noise and data with MMDiT Fewer sampling steps and bidirectional text-image interaction for stronger typography

Conditioning and Sampling Acceleration

  • Classifier-Free Guidance (CFG): Randomly drops conditioning during training and extrapolates between conditional and unconditional predictions at inference time to trade off prompt adherence against sample diversity.
  • Spatial & Style Adapters: Modules such as ControlNet and IP-Adapter inject edge maps, depth, pose, or reference subjects without retraining the base generator.
  • Fast Solvers & Step Distillation: While early DDPM required hundreds of steps, ODE solvers (DDIM, DPM-Solver) and consistency/step distillation reduce generation to 4–20 steps.

Limitations

  • Iterative Latency: Multi-step denoising still incurs higher per-sample latency than single-pass autoregressive or feed-forward models, especially for long videos.
  • Physical & Temporal Consistency: Video diffusion models still exhibit artifacts such as impossible physics, hand geometry errors, and long-horizon drift.
  • Provenance & Attribution: Training-data provenance, likeness rights, and verifiable watermarking (such as C2PA metadata) remain active governance requirements.

Further Reading