BitDecoding-FP4 Delivers Up to 3× Peak Throughput with Native FP4 KV Caches
Summary:OpenBitSys presents BitDecoding-FP4, an algorithm-system co-design for native FP4 key-value cache decoding. It combines dual-residual query quantization, underflow-aware probability scaling, and outlier-aware key quantization with a transposed softmax, fused probability quantization, stacked query MMA, and paged KV-cache integration. On NVIDIA B300 systems, the implementation reaches 2.4–3.0× the peak serving throughput of a BF16 KV-cache baseline with Qwen3-4B and Qwen3-30B-A3B. The reported a
Core Result
OpenBitSys's BitDecoding-FP4 does more than compress a BF16 KV cache into 4-bit storage. It addresses both quantization error and attention-kernel bottlenecks. On the algorithmic side, it corrects query residuals, Softmax probability underflow, and key outliers. On the systems side, transposed Softmax, fused probability quantization, stacked query MMA, and paged KV-cache integration reduce high-precision intermediate traffic.
On NVIDIA B300, the implementation raises peak serving throughput to 2.4–3.0× that of a BF16 KV-cache baseline for Qwen3-4B and Qwen3-30B-A3B. The reported NVFP4 accuracy gap remains within 0.5 percentage points of BF16, although cross-platform latency and efficiency remain unverified.
Why FP4 KV Caches Are Not a Drop-In Change
Autoregressive decoding reads the complete historical K/V state for every generated token. As context length grows, KV capacity and memory bandwidth become dominant constraints. For conventional multi-head or grouped-query attention, cache size can be approximated as:
M_KV ≈ 2 × L × H_kv × S × d_head × b / 8
Here, L is the number of layers, S is the sequence length, and H_kv is the number of KV heads. BF16 uses 2 bytes per scalar, while FP4 uses 0.5 bytes, theoretically reducing the payload to one quarter. Block scales, metadata, paging overhead, and alignment prevent the realized saving from equaling that ideal.
The computation can then become limited by Softmax, probability scaling, or layout conversion. In long contexts, a few probabilities rounded to zero, an outlier key dimension, or a distorted query logit can alter token ranking even when average accuracy appears stable. BitDecoding-FP4 is designed around these coupled failure modes.
Three Algorithmic Corrections
1. Dual-Residual Query Quantization
Quantizing the main query representation can distort the ordering of query-key logits. This matters because one query may be compared with thousands of keys, allowing small numerical errors to propagate into the attention distribution. The method splits the query into a primary FP4 tensor and an FP4 residual tensor, conceptually q ≈ q₀ + Δq, with the residual preserving information lost by the main quantization path.
The paper reports a reduction in query-logit error from 60% to 1%. The approach adds query-side tensors and computation, but the transient query state is much smaller than a long-lived KV cache, so the overhead can be amortized by lower cache traffic and higher achievable concurrency.
2. Underflow-Aware Probability Scaling
Softmax produces many extremely small probabilities. When these probabilities are quantized with E4M3 scales, the limited exponent range can round long-context tail probabilities to zero. That silently truncates the effective attention range.
Underflow-Aware Probability Scaling modifies the probability-scaling path to prevent nonzero values from disappearing during scale processing. Its primary target is not average accuracy, but tail behavior that can be hidden by aggregate metrics.
3. Outlier-Aware Key Quantization
Token-wise scaling can be dominated by a small number of outlier dimensions. One outlier raises the scale ceiling and reduces the effective precision available to all remaining dimensions. Outlier-Aware Key Quantization combines static smoothing, Hadamard rotation, and scale search.
The Hadamard transform spreads outlier energy across coordinates, while scale search finds a more suitable representation for the key distribution. The method trades preprocessing overhead for more stable FP4 quantization, but its sensitivity to model distribution and the cost of parameter or scale selection still need to be measured from the released implementation.
System Path: Removing Softmax and Intermediate Round Trips
Although production inference stacks often use fused attention kernels, Softmax, probability scaling, and layout conversion can remain distinct bottlenecks. BitDecoding-FP4 changes the score and probability layout through transposed Softmax, then fuses probability scaling with FP4 quantization inside the attention data path. This avoids writing intermediate tensors to high-bandwidth memory and reading them back.
Stacked Query MMA presents multiple query blocks in a matrix shape that better matches Tensor Core execution, allowing the same paged K/V data to serve more queries and amortizing cache reads. Paged KV storage keeps K/V in non-contiguous memory blocks, avoiding a repacking step solely to satisfy the quantized execution path.
The speedup comes from two sources: FP4 reduces cache payload and bandwidth demand by roughly 4× at the element level, while the fused kernels lower Softmax and quantization overhead. The corresponding costs are greater dependence on native FP4 Tensor Cores, block-scale management, template specialization, and inference-runtime integration.
Measured Results and Evaluation Boundaries
| Item | Reported result | Engineering interpretation | |---|---:|---| | NVFP4 accuracy | Within 0.5 percentage points of BF16 KV | The reported loss is deployable, but the aggregation metric is unspecified | | MXFP4 accuracy | Within 1 percentage point of BF16 KV | The general microscaling format has slightly greater degradation | | RL rollouts | Reported to track BF16 KV | The evaluation extends beyond conventional question answering | | Architecture coverage | Dense, MoE, GQA, and MLA | The approach targets several distinct KV-cache layouts | | B300 end-to-end throughput | 2.4–3.0× the BF16 KV baseline | Bandwidth savings translate into meaningful serving gains | | Models | Qwen3-4B and Qwen3-30B-A3B | Both dense and MoE inference are covered, but the model set remains small |
BF16 KV is a sensible high-precision reference. However, the supplied abstract does not provide matched comparisons against FP8, INT4, other FP4 KV implementations, or alternative attention kernels. It also omits batch size, input and output lengths, concurrency, scheduler settings, and the allocation between cache compression and kernel-level gains.
Peak throughput cannot substitute for TTFT, TPOT, p95 or p99 latency, cost per request, or tokens per watt. The measurements also rely on NVIDIA B300 hardware with native FP4 acceleration, so the result cannot be extrapolated directly to H100, H200, or GPUs without native FP4 Tensor Cores. The collected material does not state the paper's peer-review status, statistical variance, or a complete ablation of every algorithmic and kernel-level contribution.
Engineering Trade-offs
- The throughput gain has identifiable sources: KV payload and bandwidth fall by roughly 4× at the element level, while fused kernels reduce Softmax overhead. Ablation is still needed to separate these effects.
- Long contexts benefit most: the longer the KV state, the more likely memory bandwidth dominates execution time. For short requests, quantization and scheduling overhead may erase the advantage.
- Quality risk moves toward the tails: query logits, tiny Softmax probabilities, and key outliers can affect long-tail tasks even when mean accuracy remains stable.
- Hardware dependence is material: native FP4 Tensor Cores are important to reproducing the reported speedup. Software-emulated FP4 may deliver much smaller gains on other accelerators.
- Operational complexity increases: quantized KV blocks require scale and metadata handling, while paging allocators, cache writers, and inference frameworks must be updated. A BF16 fallback should remain available.
- Reproducibility still requires auditing: public code is a positive signal, but adopters should inspect licensing, hardware dependencies, test coverage, and the completeness of the serving integration.
Deployment Guidance
- Target workloads that are genuinely KV-bandwidth-bound: Measure long-context frequency, average cache length, concurrency, TTFT, and TPOT before introducing FP4 KV.
- Reproduce the full baseline on target hardware: Compare BF16 KV, the existing FP8 or INT4 path, and BitDecoding-FP4 using identical weight precision, scheduler, batch settings, context lengths, and output lengths.
- Start with NVFP4: The summary reports lower accuracy degradation for NVFP4 than MXFP4. MXFP4 is more useful as a cross-vendor format option, not the default precision path.
- Test long-tail quality explicitly: Cover long-document retrieval, multi-turn reasoning, code understanding, RL rollouts, and MoE models, recording task accuracy, output degradation, and rejection rates.
- Measure latency and energy, not just peak throughput: Report p50, p95, and p99 latency, GPU-seconds per request, memory consumption, and tokens per watt.
- Use canaries and precision fallbacks: Validate scales, workload drift, and outliers in shadow traffic before gradual rollout, while retaining per-request BF16 fallback for accuracy-sensitive workloads.
The implementation is available at https://github.com/OpenBitSys/BitDecoding-FP4. It is worth benchmarking for Blackwell clusters that need lower KV-cache capacity and higher long-context throughput. Whether it should replace an existing serving stack, however, depends on quality, tail latency, energy efficiency, and migration cost under a unified production workload.
— END —Source: Hugging Face Daily Papers (2026-11-06)
Tags: #FP4 Quantization #KV Cache #LLM Inference #GQA and MLA #NVIDIA Blackwell
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments