ZICQ
中 Log in / Sign up
Newsroom LLMs & Foundation Models #KV Cache #GPU Optimization #AI System Architecture #Caching Strategies #Resource Management

KV Cache Optimization Research: Storage Strategies Across GPU, CPU, and SSD

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 9 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:This research investigates KV cache placement policies across GPU, CPU, and SSD to address the scarcity of GPU high-bandwidth memory and improve AI task performance. Using a discrete event simulator calibrated against a random forest execution time predictor, the study compares recency, reuse frequency, predicted reuse, and EWMA predictors with prefetch lookahead across chat, agent, and document question answering workloads. Results show that tiered caching supports 73.02 times more concurrent s


Background and Motivation

GPU high-bandwidth memory (HBM) is scarce and expensive, and KV caches, such as those used in chats, agent loops, and document question answering, consume a significant portion of this resource. Existing systems like Mooncake, LMCache, FlexGen, InfiniGen, and AttentionStore extend GPU memory with CPU DRAM and SSD, but optimizing cache placement policies remains a challenge.

Methodology

The study employs a discrete event simulator spanning GPU HBM, CPU DRAM, and SSD, calibrated against a random forest execution time predictor. It compares recency, reuse frequency, predicted reuse, and EWMA predictors with prefetch lookahead across chat, agent, and document question answering workloads.

Key Findings

  1. Advantages of Tiered Caching: Tiered caching supports 73.02 times more concurrent sessions per GPU and reduces the cost per session by 62.04 times.
  2. Impact of Placement Policies: Placement policies have a limited effect on decode throughput but significantly affect PCIe migration traffic and time to the first token.
  3. Policy Comparison: Reuse frequency performs best for agents and document question answering, while recency reduces migration traffic by 2.30 times for chat.
  4. Prefetching Effectiveness: Prefetching does not justify its bandwidth cost and, in some cases, increases bandwidth consumption.

Industry Implications and Recommendations

  1. Developer Recommendations: Optimize cache strategies for specific workloads rather than relying on generic prefetching or prediction methods.
  2. Hardware Optimization Directions: Future hardware designs should incorporate smarter cache management mechanisms to reduce PCIe migration traffic and improve overall performance.
  3. AI System Design: AI system developers should pay attention to the impact of cache strategies on task performance, especially in resource-constrained environments.

Source: ArXiv AI (cs.AI) (2026-09-16)

— END —

Tags: #KV Cache #GPU Optimization #AI System Architecture #Caching Strategies #Resource Management

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…