arXiv Releases Cache2LoRA: Revolutionizing Context Internalization for Enhanced LLM Efficiency

Summary:arXiv introduces Cache2LoRA, a novel method that encodes documents as low-rank parameter updates, enabling language models to answer questions without retaining the source text in their decoding context. By leveraging a reusable frozen document cache and query-driven evidence aggregation, Cache2LoRA significantly improves on existing context-to-LoRA methods, achieving a macro F1 score of 0.663 across 17 tasks, approaching the performance of full-context in-context learning (0.671). Additionally,

---

Technical Mechanism Analysis

The core innovation of Cache2LoRA lies in its method of encoding documents as low-rank parameter updates (LoRA) and combining it with a reusable frozen document cache and query-driven evidence aggregation. The process involves the following steps:

1. Document Encoding and LoRA Updates: Encoding document content into low-rank parameter updates to avoid retaining the source text during decoding, thereby reducing memory usage.
2. Reusable Frozen Document Cache: Introducing a reusable frozen document cache to store encoded representations of frequently used documents, reducing the overhead of repeated encoding.
3. Query-Driven Evidence Aggregation: Using each query to aggregate evidence from the cache and generate LoRA updates. This ensures that the model can effectively utilize relevant contextual information when answering questions.
4. Multi-Style Supervised Training: Through multi-style supervised training, the model learns not only to preserve evidence but also to generate responses that comply with different instructions.

Engineering Trade-offs and Performance

Cache2LoRA optimizes in several aspects to achieve efficient and accurate inference:

• Memory and Computational Efficiency: The use of LoRA updates and frozen cache significantly reduces memory usage and computational overhead.
• Throughput Improvement: For 32K documents and 256-token outputs, Cache2LoRA achieves a decoding throughput that is 5.9 times that of full-context in-context learning (ICL).
• Accuracy: Across 17 tasks, Cache2LoRA achieves a macro F1 score of 0.663, approaching the performance of full-context in-context learning (0.671), demonstrating its ability to maintain high accuracy while improving efficiency.
• Robustness: Under long texts and diverse interference, Cache2LoRA-generated LoRA updates are more reliable than the learned KV variant and can combine more effectively with retained document KV, improving hybrid F1 and EM scores.

Developer Implementation and Deployment Recommendations

The release of Cache2LoRA provides developers with an efficient solution for LLM inference, particularly suitable for the following scenarios:

• Long Text Processing: In scenarios where long documents or frequent queries are involved, Cache2LoRA can significantly reduce memory usage and computational overhead.
• High Throughput Requirements: For applications that require high-throughput inference, such as real-time question-answering systems, the high decoding throughput of Cache2LoRA is a significant advantage.
• Multi-Task Environments: The multi-style supervised training of Cache2LoRA enables it to adapt to different task requirements, enhancing the model's generalization capabilities.

Developers can integrate Cache2LoRA into their applications by combining it with existing LLM frameworks to improve inference efficiency and accuracy. Additionally, it is recommended to adjust the cache size and update frequency according to specific needs to achieve optimal performance.

---

Source: ArXiv AI (cs.AI) (2026-10-20)

— END —

Tags: #arXiv Follow #LLMs & Foundation Models Follow #Inference Optimization Follow #LoRA Follow #Context Internalization Follow