Skip to main content
ZICQ

Wiki Technical Terms

Model Quantization

Technical Terms
Aliases: Quantization GPTQ AWQ GGUF FP8 ·2026-10-10

Model Quantization

Model Quantization is an essential deep learning systems optimization technique that reduces numerical precision across neural network parameters and activation tensors. By mapping continuous high-precision floating-point values (FP16/BF16/FP32) into discrete low-bitwidth integers (such as INT8, INT4) or reduced-precision floats (FP8, FP4), quantization dramatically slashes GPU memory footprints and multiplies arithmetic throughput with minimal degradation in model reasoning capability.

Mathematical Formulation: Uniform Affine Quantization

Uniform affine quantization projects continuous values from a bounded real range $[\alpha, \beta]$ onto a bounded integer lattice $[q_{\min}, q_{\max}]$:

1. Forward Quantization Operator

$$q = \text{clip}\left(\left\lfloor \frac{x}{S} \right\rceil + Z, q_{\min}, q_{\max}\right)$$

where:

  • $S \in \mathbb{R}^+$: The Scale Factor, denoting the step size in continuous space corresponding to a single discrete unit:

    $$S = \frac{\beta - \alpha}{q_{\max} - q_{\min}}$$

  • $Z \in \mathbb{Z}$: The Zero-Point offset, ensuring that true real-valued zero maps precisely onto an integer without rounding skew:

    $$Z = \left\lfloor \frac{-\alpha}{S} \right\rceil$$

  • $\lfloor \cdot \rceil$: The round-to-nearest integer operator.

2. Dequantization Operator

During matrix multiply or activation fusion, discrete numbers are reconstructed to approximate floating-point tensors:

$$\hat{x} = S \cdot (q - Z)$$

3. Symmetric vs. Asymmetric Quantization

  • Symmetric Quantization ($Z = 0$): Restricts the dynamic range symmetrically around origin $[-\max(|\alpha|, |\beta|), \max(|\alpha|, |\beta|)]$. Eliminating zero-point offsets enables hardware Tensor Cores to execute integer matrix multiplications without subtraction compensation, maximizing execution speed.
  • Asymmetric Quantization ($Z \neq 0$): Exploits the full dynamic spectrum of the discrete range, optimal for strictly positive activations (such as ReLU/GELU), though requiring slight runtime compensation in lower-level compute kernels.

Quantization Granularity

  1. Per-Tensor: Shares a single scale $S$ and zero-point $Z$ across the entire parameter matrix. Storage overhead is minimal, but outliers disproportionately dilute precision across normal weights.
  2. Per-Channel / Per-Token: Assigns dedicated scaling parameters per individual output channel or token vector, isolating localized cross-channel magnitude disparities.
  3. Group-wise / Block-wise: Segments channels into fine-grained blocks (e.g., groups of 64 or 128 consecutive weights sharing a scale). This represents the current gold standard for 4-bit algorithms (AWQ, GPTQ) to prevent catastrophic quality regression.

Paradigm Comparison: PTQ vs. QAT

  • Post-Training Quantization (PTQ): Compresses frozen pre-trained checkpoints offline using a tiny calibration set of 128 to 512 unlabeled domain sequences. Because it avoids backpropagation and converges within minutes, PTQ represents the predominant production approach for foundation models.
  • Quantization-Aware Training (QAT): Injects simulated quantization operators (Fake Quantization) into the computational graph during fine-tuning. By steering gradients with the Straight-Through Estimator (STE), the network dynamically adapts to discrete rounding error, significantly outperforming PTQ in sub-4-bit regimes.

The Outlier Problem in Large Models & Breakthrough Algorithms

Beyond the 6.7B parameter threshold, Transformer architectures spontaneously exhibit emergent activation outliers: a minute subset of hidden feature dimensions erupt with magnitudes 20x to 100x larger than standard features. Standard quantization flattens the resolution of remaining channels, inducing catastrophic failure.

Several breakthrough algorithms resolve this issue:

1. SmoothQuant

Leverages mathematical equivalence to migrate sharp activation outliers into the weight matrix prior to quantization:

$$Y = (X \cdot \text{diag}(s)^{-1}) \cdot (\text{diag}(s) \cdot W)$$

By redistributing dynamic ranges between activations and weights, SmoothQuant enables seamless 8-bit integer matrix multiplications across both activations and weights (W8A8).

2. AWQ (Activation-aware Weight Quantization)

Observes that parameter sensitivity is not uniform: roughly 1% of salient weights that interact directly with peak activation channels dominate overall model perplexity. By selectively preserving or per-channel scaling this 1% subset while quantizing the remainder to INT4, AWQ prevents quality degradation in compact models.

3. GPTQ (Optimal Brain Surgeon Quantization)

Formulates quantization as a column-wise quadratic optimization problem. As each column is quantized, it leverages the inverse Hessian matrix to dynamically adjust remaining unquantized weights, compensating for quantization errors with mathematical rigor.

Deployment Format Matrix

Format Weight Precision Activation Precision Target Hardware Engineering Notes
FP8 (E4M3/E5M2) 8-bit Float 8-bit Float NVIDIA Hopper, Blackwell, Ada Cloud production standard; native hardware tensor acceleration with negligible loss
AWQ 4-bit Integer 16-bit Float (W4A16) Modern NVIDIA GPUs Preferred for high-throughput production serving with state-of-the-art accuracy retention
GPTQ 4-bit Integer 16-bit Float (W4A16) Standard GPUs / vLLM Proven, highly stable offline 4-bit weight compression format
GGUF (K-Quants) 2~8-bit Mixed 16-bit Float Apple Silicon, x86 CPU, Edge The backbone of llama.cpp; excels at heterogenous CPU/GPU memory offloading

Further Reading