Skip to main content
ZICQ

Wiki Infrastructure

vLLM

Infrastructure
Aliases: vLLM PagedAttention High-throughput inference engine ·2026-10-10

vLLM

vLLM is an open-source, high-throughput, low-latency LLM serving and inference engine developed by UC Berkeley's RISELab (published at SOSP 2023). By pioneering the PagedAttention algorithm alongside iteration-level scheduling, vLLM resolved the severe memory fragmentation and dynamic batching bottlenecks inherent to naive LLM serving, establishing itself as the de facto AI infrastructure standard.


1. Core Breakthrough: The PagedAttention Architecture

In naive serving frameworks (such as early Hugging Face Transformers implementations), the KV cache (Key-Value Cache) must be statically allocated in contiguous GPU memory. Because prompt lengths vary and generation outputs cannot be predicted upfront, engines pre-allocate fixed memory buffers corresponding to the maximum sequence ceiling (e.g., 4096 or 8192 tokens), triggering catastrophic GPU memory waste:

  • Internal Fragmentation: A request generating only 100 tokens occupies thousands of unused slots that cannot be reclaimed.
  • External Fragmentation: Memory allocators fail to accommodate incoming requests due to a lack of contiguous physical chunks.
  • Waste Exceeding 60%–80%: GPU VRAM is starved by idle allocations, severely restricting concurrency batch sizes.
Naive Contiguous Allocation (Severe Fragmentation & Waste)
[ Req A (Used 30%) | Unused 70% ] [ Req B (Used 15%) | Unused 85% ]

vLLM PagedAttention (Virtual Memory Paging Analogy)
Logical KV Blocks ─────► Block Table ─────► Non-contiguous Physical GPU Pages
[ Block 0 ]               Page 12           [ Page 12 (16 tokens) ]
[ Block 1 ]               Page 4            [ Page 4  (16 tokens) ]
[ Block 2 ]               Page 89           [ Page 89 (16 tokens) ]

The Virtual Memory Paging Analogy

PagedAttention directly adapts virtual memory paging principles from operating systems:

  1. Physical Blocks: Partitions Key and Value tensors for each attention layer into fixed-size physical pages (typically 16 or 32 tokens per block).
  2. Dynamic Block Tables: Maintains a lightweight logical mapping table per stream. Generated tokens claim non-contiguous physical blocks dynamically on demand.
  3. Memory Waste Under 4%: Fragmentation is confined solely to the tail end of the final uncompleted block per sequence, elevating VRAM utilization to near 100% and boosting serving throughput by $2\times$ to $4\times$.
  4. Copy-on-Write (CoW) & Prefix Caching: When concurrent requests share identical system instructions, few-shot examples, or branch during beam search, their logical tables point to shared physical memory pages. Pages are duplicated only when a branch produces diverging tokens, reducing prompt context reuse cost to near zero.

2. Dynamic Scheduling: Continuous Batching & Chunked Prefill

2.1 Continuous Batching (Iteration-Level Scheduling)

Traditional batching forces the scheduler to wait until the slowest long-generation request completes before accepting new incoming queries.

  • vLLM deploys iteration-level scheduling: at every single token generation cycle, completed requests immediately release their memory blocks and exit, while queued requests are spliced directly into the active batch without pipeline bubbles.

2.2 Chunked Prefill

Under long-context workloads, massive prompt prefill steps monopolize GPU streaming multiprocessors, causing severe inter-token latency spikes (ITL jitters) for active decoding streams.

  • vLLM partitions long prompts into discrete chunks (e.g., 512 tokens), interleaving prefill slices with regular decode operations to guarantee smooth output cadence.

3. Distributed Parallelism & Hardware Kernel Ecosystem

vLLM is architected for large-scale production multi-GPU clusters:

  • Tensor Parallelism (TP): Exploits Megatron-LM tensor partitioning across multi-GPU single-node configurations via high-speed NVLink All-Reduce primitives.
  • Pipeline Parallelism (PP): Partitions frontier 70B and 405B models across multi-node fabrics.
  • Quantization Support: Native integration with FP8 (E4M3/E5M2), AWQ, GPTQ, SqueezeLLM, and INT4, highly tuned for NVIDIA Blackwell, Hopper, and Ada Lovelace architectures.
  • Accelerated Kernels: Integrates FlashAttention-2, FlashAttention-3, and FlashInfer, leveraging CUDA Graphs during decode phases to eliminate CPU dispatch overheads.

4. Production Serving & Tuning Guidelines

Deploy an OpenAI-compatible endpoint with production flags:

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.3-70B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.92 \
    --max-model-len 32768 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --dtype bfloat16

Essential Parameter Cheatsheet:

  • --gpu-memory-utilization: Sets maximum allocated VRAM proportion (default 0.90; recommended 0.92–0.95 in dedicated production). Over-allocation risks CUDA Graph capture failure.
  • --enable-prefix-caching: Highly recommended. Drastically reduces Time To First Token (TTFT) across agent loops, recurring system prompts, and multi-turn chats.
  • --enable-chunked-prefill: Crucial for mixed workloads with concurrent long prompts and interactive chats.
  • --speculative-model: Couples with compact draft models to activate speculative decoding, accelerating single-stream generation by $1.5\times$ to $2.5\times$.
  • Guided Structured Outputs: Native Outlines and XGrammar integration guarantees lossless, regex-, and JSON Schema-constrained decoding with sub-millisecond overhead.