Fathom: Sparse Decoding for KV Cache Optimization Released
Summary:Fathom is a novel sparse decoding technique that optimizes KV cache access efficiency by intelligently determining the read depth per query. It significantly enhances GPU decoding speed by dynamically deciding the number of bits to read for each key channel in million-token sessions. On the Qwen3-8B model, Fathom achieves a 1.67x faster decode step compared to traditional methods and reads 18% fewer bytes in the same GPU time, with lower attention error. This technology is particularly beneficia
Background and Challenges
In handling large-scale agentic sessions, AI models need to process requests with millions of tokens, posing significant challenges to the efficiency of KV cache (key-value cache) and index access. Traditional methods, which scan all n keys to determine the top-k step priority, become a performance bottleneck, limiting GPU decoding speed.
Core Innovation of Fathom
Fathom introduces a sparse decoding technique for KV cache optimization with the following key features:
- Per-Query Read Depth Optimization: Fathom allows each query to intelligently decide the number of bits to read for each key channel, using a reverse water-filling algorithm to allocate bit budgets over the variance-weighted importance of the channels.
- 4-bit K Cache Storage: The technique stores the 4-bit K cache as bit planes in channel-major order, making the prefix of t planes exactly the channel's t-bit quantizer.
- Performance Improvement: On the Qwen3-8B model, Fathom achieves a 1.67x faster decode step compared to traditional methods and reads 18% fewer bytes in the same GPU time, with lower attention error in six out of seven model and context settings.
Experimental Results and Advantages
- Faster Decoding: Fathom's decoding speed is 1.67x faster than traditional methods on the Qwen3-8B model.
- Reduced Byte Reads: Fathom reads 18% fewer bytes in the same GPU time.
- Lower Attention Error: Fathom achieves lower attention error in six out of seven model and context settings.
Use Cases and Future Prospects
Fathom is particularly suitable for applications that require processing large-scale agentic sessions and complex tasks, such as multi-agent collaboration and real-time dialogue systems. The release of this technology provides a new approach to improving AI inference efficiency and is expected to be widely adopted in more AI models and application scenarios in the future.
Developer Recommendations
Developers can integrate Fathom into existing AI inference pipelines to improve KV cache access efficiency. Additionally, it is recommended to pay attention to Fathom's performance on different models and tasks and optimize it according to specific needs.
— END —Source: Hugging Face Daily Papers (2026-09-15)
Tags: #Fathom #KV Cache #Sparse Decoding #GPU Optimization #AI Inference
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments