llama.cpp Adds FP8 KV Cache to Reduce Local Inference Memory
By Mr.Xu Community Post
Published:
Summary:The llama.cpp community has introduced support for an FP8 KV cache, targeting the memory cost of autoregressive inference, which grows with context length and concurrency. By storing key and value states in 8-bit floating point, the feature could reduce memory pressure for long-context and multi-user workloads. The practical benefit will depend on efficient hardware support for FP8 conversion and on how quantization affects output quality, latency, and cache management. The collected information
Core Change
The llama.cpp community has added an FP8 KV cache implementation. The KV cache stores key and value states reused during autoregressive generation, and its memory footprint generally grows approximately linearly with batch size, context length, and layer count. Compressing the cache format can therefore release more usable memory than further compressing model weights in long-context or concurrent local inference.
The entity attribution is straightforward: llama.cpp is maintained by ggml-org, and the FP8 KV cache is an internal inference-engine feature rather than a new model release.
Why Compress the KV Cache
A standard autoregressive decoder repeatedly reads the K/V states of historical tokens to avoid recomputing the complete prefix. For a conventional Transformer, the approximate cache size per layer is:
M_KV = 2 × B × L × H_kv × S × d_head
Here, B denotes batch size or the number of concurrent sequences, L is the number of layers, H_kv is the number of KV heads, S is sequence length, and d_head is the head dimension. The factor 2 accounts for both K and V. With grouped-query attention, H_kv is smaller than the number of attention heads, making the cache considerably lighter than full attention activations.
Converting KV states from BF16/FP16 to FP8 can approximately halve the theoretical cache footprint when all other dimensions remain unchanged and metadata and alignment overhead are ignored. The recovered capacity can be used for longer contexts, more concurrent sessions, or larger batches, potentially reducing KV-cache memory pressure.
Benefits and Error Boundaries
Compared with INT8, FP8 provides a wider dynamic range and more direct floating-point semantics, which can suit attention states with uneven numerical distributions. However, halving the bit width does not automatically halve runtime latency:
- Model weights are typically loaded before generation, whereas the KV cache is continuously read and written during every decoding step.
- If CPU, CUDA, or Metal backends lack an efficient native FP8 path, runtime conversions may cancel out bandwidth and capacity gains.
- Errors in K/V states propagate through Softmax probabilities and value-weighted aggregation. Long-context retrieval, code generation, and strict reasoning tasks may be more sensitive to these perturbations.
- Actual quality depends on scaling policy, statistical granularity, error accumulation, and whether additional rounding occurs during cache chunking or migration.
A proper evaluation should therefore compare memory usage together with time to first token, sustained decoding throughput, batch size, peak memory, and task accuracy.
Implications for Deployment Paths
On consumer GPUs, FP8 KV cache may allow users to retain longer contexts or run multiple sessions when memory capacity previously constrained local inference. In multi-GPU tensor-parallel deployments, layer- or device-sharded KV caches can also reduce per-shard pressure, although gains still depend on cache placement and memory fragmentation.
For CPU-only inference, the main benefit may be reduced memory consumption rather than higher speed. If the backend repeatedly converts among FP8, FP16, and BF16, the memory savings may be offset by conversion overhead. Apple Silicon and Metal users should inspect the actual FP8 instruction path, alignment requirements, and kernel-fusion implementation rather than inferring performance from theoretical bit width alone.
Practical Deployment Guidance
- Wait for the pull request to be merged and verify the actual command-line flags and backend support matrix in llama.cpp.
- Compare the default KV format with FP8 KV cache using the same model, context, and hardware.
- Record peak memory, prefill time, time to first token, generation tokens per second, and throughput at different batch sizes.
- Test code generation, long-context retrieval, and multi-turn conversations to detect quality degradation rather than relying only on generic perplexity.
- Keep BF16/FP16 KV cache as a fallback. High-precision workloads should not force FP8 when meaningful output drift is observed.
Current Information Boundary
The collected page provides only the title, submitter, and GitHub pull-request link. It does not include the PR description, code diff, test data, supported hardware, or merge status. What can currently be confirmed is that the llama.cpp community is progressing FP8 KV-cache support; no specific acceleration factor or lossless quantization guarantee can yet be inferred. Production adoption should follow a direct code review and reproducible end-to-end benchmark.
— END —Source: Reddit r/LocalLLaMA (2026-11-11)
Tags: #llama.cpp #FP8 #KV Cache #Local Inference #Memory Optimization
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments