ZICQ
中 Log in / Sign up
Newsroom Compute & Architecture #AWS #Distributed KV Cache #LMCache #vLLM #Agent Skills

AWS Agent Skill Operationalizes Distributed KV Caching for Self-Hosted LLMs

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:AWS has made a distributed KV caching Skill available through Agent Toolkit for AWS. Coding agents such as Kiro, Claude Code, and OpenAI Codex can use it to assess workload eligibility, estimate savings with KVTrace, configure LMCache with vLLM or SGLang, validate cache-loaded outputs, and troubleshoot RDMA transfer failures. The shared L2 tier is built on Amazon ElastiCache for Valkey and extends cache reuse across GPU servers, sessions, routing changes, and failures. AWS reports that the unde


What changes: from node-local caching to a shared L2 tier

AWS has made a distributed KV caching Skill available through Agent Toolkit for AWS. It is not a new attention or decoding algorithm. Instead, it connects a shared L2 tier built on Amazon ElastiCache for Valkey to existing vLLM and SGLang deployments, then standardizes workload evaluation, configuration, output validation, and troubleshooting through an Agent Skill.

  • Node-local L1 cache: Offers the lowest access latency, but its state is tied to one GPU server. A request routed elsewhere, a server failure, or a user returning after a long pause causes the state to be lost.
  • Shared L2 cache: Stores reusable KV state across GPU instances, allowing another server to resume long-running sessions or continue work after a routing change or failure.
  • LMCache connector: Locates, transfers, and loads KV blocks between the inference engine and shared cache. EFA with RDMA Write reduces the overhead of moving large state blocks.

The primary target is cold Prefill in long-context agent workloads. If a user returns to a legal-assistant thread after an hour, and the next request lands on a different GPU, the server normally has to recompute the historical prefix. A shared cache lets it resume that prefix instead.

Why it works—and what it does not eliminate

For a prompt of S tokens with a reusable prefix of S₀ tokens, a cache hit avoids projecting and largely recomputing the old prefix. It does not eliminate all attention: newly generated queries still have to attend over the complete historical KV state.

KV memory can be approximated as:

M_KV ≈ 2 × L × H_kv × D_head × S × b

Here, L is the number of layers, H_kv is the actual KV-head count, D_head is the head dimension, and b is bytes per element. For grouped-query attention, H_kv rather than the query-head count should be used. BF16 usually requires about two bytes per element, while FP8 and grouped quantization add scale metadata. Tensor parallelism shards the aggregate memory across ranks, and block alignment introduces additional overhead.

A useful cost comparison is:

T_hit ≈ T_lookup + T_RDMA + T_prefill(ΔS) + T_attention(ΔS, S)

T_recompute ≈ T_prefill(S)

where ΔS is the number of new tokens. The shared tier wins only when the KV transfer plus lookup overhead is lower than the cost of recomputing the old prefix. A short prompt may be cheaper to recompute on the GPU because the transferred block is small and fixed network, index, and validation costs dominate. Long, frequently reused prefixes and workload migration provide a much stronger case.

The main latency benefit is lower TTFT. It does not proportionally shorten per-token Decode time: decoding still reads KV state and executes attention for every generated token, while the shared tier adds an upfront data-transfer stage.

The Agent Skill turns deployment into an auditable workflow

The Skill supports coding agents that implement Agent Skills, including Kiro, Claude Code, and OpenAI Codex. Its workflow covers five operational stages.

  1. Eligibility check: The agent first uses a read-only describe-instances call to inspect the GPU fleet and count EFA interfaces. It then asks for the model, inference engine, Tensor Parallel size, KV data type, server price, and prompt-repetition characteristics. If local GPU prefix caching already provides a high hit rate with strong session affinity, the Skill explains that a shared cache may add little value.
  2. Savings estimation: KVTrace records cache events from vLLM or SGLang and replays them offline against candidate cache capacities. By default, it does not record token IDs or prompt text; traces retain block hashes and token counts. Because hashes are derived from content, sensitive prompts should use --anonymize-key-file to avoid correlation-based identification.
  3. Cluster configuration: The Skill adds LMCache to an existing vLLM or SGLang deployment, including vLLM workers under llm-d and Ray Serve. Before creating resources, it requests explicit confirmation and configures EFA security groups, an ElastiCache for Valkey 9.1 cluster, eviction policies, a dedicated LMCache user, and Secrets Manager credentials.
  4. Correctness validation: All inference servers must agree on the hash and seed mechanism, KV data type, and compatible TP layout. Before production traffic, the Skill flushes local tiers and compares two cache-loaded runs. This catches a particularly dangerous failure mode in which hit counters look healthy while layout mismatches produce incorrect outputs.
  5. RDMA troubleshooting: If BLOB.RDMA_HELLO succeeds while rdma_write_errors continues to rise, a common cause is a missing self-referencing EFA rule between security-group members. The control-plane exchange can succeed even though actual data writes are blocked.

Where the savings come from

AWS reports that the underlying distributed caching architecture can reduce inference costs by up to 40% and TTFT by up to 80% in suitable workloads. These are upper bounds rather than default outcomes for any model, prompt length, or traffic pattern.

Net economics can be viewed as:

Avoided Prefill GPU cost + value of released capacity - cache cost - RDMA cost - operational overhead

The main variables are:

  • Prefix length and repetition: Short or rarely repeated prompts are unlikely to cover the cost of fetching cached state.
  • Hit quality: A useful hit must cover a long prefix that would otherwise be recomputed, not merely a small common suffix.
  • Local-cache baseline: If GPU-resident prefix caching already handles most reuse, the marginal benefit of a shared tier is limited.
  • Routing behavior: Server changes, failover, and resumption after idle periods are where distributed caching has the clearest advantage.
  • Network and eviction: KV transfer bandwidth, capacity-driven eviction, and subsequent recomputation all reduce the theoretical benefit.

Freed GPU time does not automatically reduce the bill. The saving becomes financial only if the fleet can be reduced or the same fleet can serve more traffic. If utilization remains low, distributed caching may mainly improve waiting time rather than infrastructure spend.

How to interpret the benchmark claims

The legal-assistant example uses eight p5 instances, a 64K-token context, and a sizing recommendation of two cache.i8ge.48xlarge cache nodes. It is a configuration-estimation scenario, not a complete controlled benchmark. The illustrated KVTrace report also uses synthetic demo traffic.

A production evaluation still needs:

  • Model names, quantization format, TP degree, GPU type, and concurrency level;
  • Prompt-length distribution, prefix-reuse distribution, real hit rate, and hit-token length;
  • TTFT at p50, p95, and p99, together with Decode throughput and end-to-end latency;
  • RDMA bandwidth, transfer-error rate, eviction rate, and output consistency after cache loads;
  • Net-cost curves against local prefix caching, no shared cache, and several cache capacities.

The Skill does not create the acceleration itself. Its value is standardizing estimation, deployment, verification, and troubleshooting. The actual performance gain still comes from reusing KV state and skipping redundant Prefill.

Deployment playbook

  1. Establish a baseline: Record new-token counts, repeated-prefix ratios, local prefix-cache hit rates, Prefill time, TTFT percentiles, Decode throughput, route-change frequency, and failover events.
  2. Check the compatibility matrix: Current prerequisites include EFA-capable RDMA Write GPUs such as p5; LMCache 0.5.8rc1 or later; vLLM 0.11.1 or later, or SGLang 0.5.13 or later; and ElastiCache for Valkey 9.1 with the appropriate cache nodes. Because the listed LMCache build is a release candidate, run upgrade regressions before production adoption.
  3. Canary in one Availability Zone: Start with representative traffic, collect KVTrace events, and compare directly against the existing local prefix-cache baseline. Do not size the cache from prompt length alone or from an average hit rate without hit-length distribution.
  4. Make configuration identity a release gate: Align model version, TP sharding, KV dtype, block layout, hash, and seed. Any change that alters KV byte layout or positional-encoding behavior should invalidate old cache entries.
  5. Test cross-server recovery: Flush local tiers, send the same request to another GPU, and compare outputs, TTFT, and hit logs. Then exercise GPU failure and route migration.
  6. Isolate multi-tenant exposure: Shared caches can reveal whether another tenant submitted the same prefix through response-time behavior. Use separate caches where strong isolation is required; otherwise include a tenant identifier in the KV hash and continue monitoring timing side channels.
  7. Harden networking and permissions: Permit only required EFA traffic and verify self-referencing security-group rules. Give LMCache a dedicated least-privilege account, store its credential in Secrets Manager, and retain human approval for resource creation, modification, and deletion.
  8. Size against realized savings: KVTrace's smallest capacity reaching roughly 90% of the modeled savings ceiling can be a starting point. Final sizing should include hit-token length, eviction rate, network utilization, and cost per effective cached token.

Engineering verdict

The meaningful release is not a faster inference algorithm but an operational interface for distributed KV caching. It is most relevant to long-running legal assistants, resumable coding agents, multi-shift RAG applications, and deployments where failures routinely move work to another GPU server.

It is not a universal acceleration switch. Short prompts, low reuse, sticky routing, or a high local prefix-cache hit rate favor keeping the existing architecture. For workloads with long reusable prefixes, the payoff comes from skipping old-prefix Prefill, while KVTrace-based workload validation, strict cache-key and layout consistency, and a stable RDMA path determine whether that payoff survives contact with production.


Source: AWS Machine Learning Blog (2026-11-21)

— END —

Tags: #AWS #Distributed KV Cache #LMCache #vLLM #Agent Skills

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…