Skip to main content
ZICQ

Wiki Infrastructure

llama.cpp & GGUF

Infrastructure
Aliases: llama.cpp GGUF ggml ·2026-11-18

llama.cpp & GGUF

llama.cpp is an open-source C/C++ LLM inference engine initiated by Georgi Gerganov on top of the ggml tensor library. Free from heavy Python runtimes or framework dependencies, it runs quantized models across CPUs, Apple Silicon (Metal), NVIDIA CUDA, AMD ROCm, and Vulkan—and established the GGUF single-file model packaging standard that powers local tools such as Ollama and LM Studio.

Why llama.cpp Anchors Local Inference

  • Minimal Runtime Footprint: Compiles to compact native binaries or libraries suitable for CLI workflows, desktop apps, and offline/air-gapped edge deployments.
  • Hybrid CPU/GPU Layer Offloading: When a model exceeds available VRAM, -ngl lets engineers offload as many Transformer layers as fit onto the GPU while streaming remaining layers from system RAM rather than failing with out-of-memory errors.
  • Memory-Mapped Loading (mmap): Together with GGUF, weights map directly through OS page caches, enabling fast startup and shared read-only pages.

GGUF Format and Quantization Tiers

GGUF (successor to GGML) bundles architecture hyperparameters, tokenizer vocabulary/merges, tensors, and quantization metadata into one self-describing binary file with backward-compatible metadata keys.

Quantization Approx. Bits/Weight Practical Trade-off
Q8_0 8-bit Near-lossless quality when memory headroom is plentiful
Q5_K_M / Q6_K ~5–6 bit High fidelity close to FP16 for coding and multi-step reasoning
Q4_K_M ~4.5 bit Widely used default balancing memory savings and output quality
IQ2_XS / Q3_K_S 2–3 bit Extreme memory compression (with importance-matrix calibration); noticeable quality drop on complex tasks

Running llama-server

Beyond llama-cli, llama.cpp ships llama-server with an OpenAI-compatible HTTP endpoint, continuous batching, and prompt-prefix caching:

llama-server -m ./model-q4_k_m.gguf \
  -c 32768 \
  -ngl 99 \
  --port 8080

Key tuning flags include -c (context length, which scales KV cache memory), -ngl (GPU layer offload count), and -fa (FlashAttention kernel acceleration).

Choosing Between llama.cpp and Server Engines

Dimension llama.cpp / GGUF vLLM / SGLang
Strength Laptops, Apple unified memory, hybrid CPU+GPU, zero-Python packaging High-concurrency multi-GPU datacenter serving with PagedAttention
Trade-off Lower batched multi-tenant throughput than datacenter engines Requires Linux and modern discrete GPUs
Best Fit Local coding assistants, offline edge appliances, single-user workstations Production API gateways and high-throughput agent backends

Further Reading