Model Inference
Model inference (in enterprise production environments termed LLM Serving) is the systems engineering practice of deploying trained or fine-tuned Large Language Models (LLMs) across high-performance compute clusters to expose low-latency, high-concurrency, high-throughput text generation APIs.
Unlike traditional stateless Web services or convolutional neural networks (CNNs), serving autoregressive LLMs entails massive resident memory requirements (tens to hundreds of gigabytes), dynamic variable-length inputs/outputs, and two distinct computational regimes governed by conflicting hardware bottlenecks.
The Roofline Model: Prefill vs. Decode Phases
Autoregressive inference is not a uniform workload. At the hardware layer, it splits into two phases governed by opposing performance ceilings:
LLM Inference Dual-Phase Profile
│
┌──────────────────────────────┴──────────────────────────────┐
▼ ▼
[Prefill Phase (Prompt Processing)] [Decode Phase (Token Generation)]
Metric: Time To First Token (TTFT) Metric: Time Per Output Token (TPOT)
Workload: Evaluates all L_prompt input tokens in parallel Workload: Sequential step-by-step; 1 token emitted per pass
Kernel: GEMM (Matrix-Matrix Multiply), high arithmetic density Kernel: GEMV (Matrix-Vector Multiply), low arithmetic density
Constraint: Compute-Bound Constraint: Memory Bandwidth Bound
Optimization: FlashAttention Tiling Optimization: PagedAttention, KV Compression, Speculative Decoding
1. Prefill Phase (Time To First Token - TTFT)
- Mechanism: The engine ingests user prompt tokens ($L_{\text{prompt}}$) concurrently in a single forward pass, calculating dense self-attention and populating the initial KV cache;
- Roofline Profile: Parallel token execution makes this stage compute-bound. GPU Tensor Cores operate at high saturation. Latency depends on hardware peak floating-point throughput (TFLOPs), accelerated via FlashAttention.
2. Decode Phase (Time Per Output Token - TPOT)
- Mechanism: Enters an autoregressive loop. Each step processes only the newly generated token combined with the historical KV cache, sampling the subsequent token;
- Roofline Profile: At small-to-moderate batch sizes, single-token generation consists of vector-matrix multiplications (GEMV). To perform these modest arithmetic operations, the GPU must stream the model's entire parameter footprint from High Bandwidth Memory (HBM) to compute registers for every single token. Hardware compute utilization frequently hovers below $5%$. This stage is strictly memory-bandwidth bound, with throughput governed by HBM read rates (TB/s).
Core Systems Innovations in Modern Inference Engines
Production engines like vLLM, TensorRT-LLM, and SGLang achieve high throughput through four core systems primitives:
┌────────────────────────────────────────────────────────────────────────┐
│ Modern LLM Serving Systems Architecture │
│ │
│ [Scheduling] ──► Continuous Batching (Iteration-level scheduling) │
│ Dynamically inserts prefills; evicts finished tokens │
│ │
│ [Memory] ──► PagedAttention (Virtual memory pagination) │
│ Allocates KV cache into block linked lists │
│ │
│ [Caching] ──► Prefix Caching (RadixAttention) │
│ Reuses system prompts and multi-turn context caches │
│ │
│ [Execution] ──► Speculative Decoding + FP8 / AWQ Quantization │
└────────────────────────────────────────────────────────────────────────┘
1. PagedAttention (Virtual Memory Pagination)
- The Traditional Bottleneck: Legacy engines pre-allocated contiguous memory pools based on the theoretical maximum sequence length. Because actual generation lengths vary, $60% \sim 80%$ of GPU memory sat idle as internal or external fragmentation;
- Paged Design: Inspired by OS virtual memory paging, PagedAttention segments KV cache tensors into fixed-size physical blocks (e.g., 16 or 32 tokens). Page tables map logical token sequences to non-contiguous physical blocks on demand. Memory fragmentation drops from $60%+$ to $<4%$, increasing serving concurrency by $2\times \sim 4\times$.
2. Continuous Batching (Iteration-Level Scheduling)
- Static Batching Defect: Because sequence lengths vary widely, traditional batches forced fast requests to idle while waiting for the longest response in the batch to complete;
- Dynamic Iteration Scheduling: At each token generation iteration, the scheduler dynamically injects newly arrived prefill requests into the batch and evicts requests hitting their End-Of-Sequence (EOS) token, increasing cluster throughput by $10\times \sim 20\times$.
3. Prefix Caching (RadixAttention)
In multi-turn chat and agentic workflows, requests frequently share identical system instructions, few-shot templates, or document contexts. RadixAttention maintains historical KV caches inside an in-memory radix tree. New requests match existing prefix branches, bypassing redundant prefill computation and reducing Time To First Token (TTFT) by $80% \sim 95%$.
4. Speculative Decoding and Quantization
Pairing large models with lightweight drafters via speculative decoding or applying FP8/AWQ quantization halves memory traffic, accelerating token generation during the memory-bound decode phase.
Distributed Serving Topologies: Tensor vs. Pipeline Parallelism
When model parameters exceed single-GPU memory capacity, distributed topologies balance latency and throughput:
| Topology | Partition Dimension | Interconnect Demand | Primary Objective | Use Case |
|---|---|---|---|---|
| Tensor Parallelism (TP) | Shards weight matrices column-wise or row-wise within each layer | Extreme (multiple All-Reduce calls per layer; requires NVLink) | Minimizes per-request latency (Latency-Oriented) | Real-time interactive applications (copilots, low-latency chat) |
| Pipeline Parallelism (PP) | Distributes sequential Transformer layers across distinct nodes | Moderate (transmits activation vectors across layer boundaries) | Maximizes aggregate throughput (Throughput-Oriented) | High-throughput offline batch processing; massive models |
| PD Disaggregation | Physically decouples prefill workers from decode workers | High network bandwidth (streams generated KV caches via RDMA) | Eliminates compute/memory contention across phases | Hyperscale centralized API serving clusters |
Production Telemetry: The SLO Triangle
Infrastructure teams monitor LLM serving across three golden metrics:
- TTFT (Time To First Token): Latency from query arrival to first streamed token, tracking prefill compute saturation;
- TPOT (Time Per Output Token): Latency between consecutive output tokens during streaming, tracking decode memory bandwidth (typically targeted under $30\sim 50\text{ ms/token}$ for human reading cadence);
- Throughput (Tokens Per Second, TPS): Aggregate tokens emitted across the cluster per second while conforming to latency SLOs, dictating infrastructure economics (cost per million tokens).
Sources and Key Literature
- Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., SOSP 2023) — Foundational vLLM and PagedAttention architecture.
- Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., OSDI 2022) — Introduction of iteration-level continuous batching.
- SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al., 2024) — RadixAttention and prefix caching runtime.
- Fast Inference from Transformers via Speculative Decoding (Leviathan et al., ICML 2023) — Lossless speculative decoding theory.