arXiv Releases Galahad-KV: Breakthrough in AI Long-Term Memory, Enhancing LLM Inference Efficiency

Summary:arXiv has released a new study on AI long-term memory, introducing an open-source package called Galahad-KV. This package saves the key-value (KV) states of large language models to local NVMe disks and reloads them during subsequent inferences, avoiding recomputation and leading to more efficient processing. Experiments show that Galahad-KV speeds up inference by 2.8x to 4.3x and reduces GPU energy consumption by 8.8x to 12.3x when handling text sequences of up to 50 million tokens, while maint

---

Core Breakthrough

The arXiv team has introduced an open-source package named Galahad-KV, aimed at addressing the computational bottlenecks faced by large language models (LLMs) when processing long texts. The key features of this package include:

• Persistent KV State Storage: Saves the model's key-value (KV) states to local NVMe disks, avoiding recomputation.
• Efficient Loading Mechanism: Ensures the integrity and accuracy of state restoration through byte-exact loading.
• Performance Improvement: Achieves a 2.8x to 4.3x speedup in inference and an 8.8x to 12.3x reduction in GPU energy consumption when handling text sequences of up to 50 million tokens.

Technical Details

Galahad-KV works by splitting the KV states of large language models into blocks of approximately 16,000 tokens and storing them encrypted on local NVMe disks. During inference, the model can load these states directly from the disk, eliminating the need for recomputation and saving computational resources and time.

Experiments were conducted using an NVIDIA H100 GPU and Gemma 4 12B and Gemma 4 31B models, testing 50,000,000 tokens of real public text data. The results showed that:

• Loading Speed: Loading a block is 2.8x to 4.3x faster than recomputation.
• Energy Reduction: GPU energy consumption is reduced by 8.8x to 12.3x.
• Accuracy: When asked about facts planted millions of tokens earlier, the 12B model gave the correct answer 82 out of 100 times, and the 31B model 98 out of 100 times.

Industry Impact

The release of Galahad-KV has the following impacts on the AI industry:

• Enhanced Long-Text Processing Efficiency: Solves the computational bottleneck problem for large models in long-text processing.
• Reduced Resource Consumption: Significantly reduces GPU energy consumption, extending hardware lifespan.
• Promotes Long-Term Memory Research: Provides a new technical path for AI long-term memory research and application.

Developer Recommendations

• Try Galahad-KV: Developers working with long texts are encouraged to try Galahad-KV to improve inference efficiency and reduce resource consumption.
• Stay Updated: Keep an eye on the arXiv team's future research developments for more innovations on long-term memory.
• Engage with the Open-Source Community: Participate in the Galahad-KV open-source community, share experiences, and contribute code.

---

Source: ArXiv NLP/LLM (cs.CL) (2026-10-09)

— END —

Tags: #arXiv Follow #Galahad-KV Follow #Long-Term Memory Follow #LLMs & Foundation Models Follow #Inference Efficiency Follow