Amazon SageMaker AI Adds KV Cache-Aware Routing for Real-Time LLM Endpoints
Summary:Amazon SageMaker AI has introduced `KV_AWARE` routing for real-time inference endpoints. Supported model containers publish KV-cache insertion and eviction events, while a managed index stores only truncated one-way hashes and instance pointers. For each request, SageMaker tokenizes the prompt, locates the instances holding the longest cached token prefix, and balances cache reuse against current instance load. The feature targets workloads where fixed-prefix routing becomes unreliable, includi
Amazon SageMaker AI now offers KV_AWARE routing for real-time inference endpoints. Instead of inferring where a cached prompt prefix might reside, the service uses cache-state events reported by the model container and sends each request to an instance that actually holds the longest reusable token prefix, subject to current load.
Routing mechanism: making the scheduler cache-aware
The existing PREFIX_AWARE strategy hashes a fixed-length request prefix: PrefixLength bytes for InvokeEndpoint, or characters for the OpenAI-compatible Chat Completions API. It requires no container feedback, but it has two operational blind spots.
First, cache state changes. When GPU memory pressure causes vLLM to evict KV-cache blocks, a fixed-prefix router can continue sending requests to the original instance even though the cache is gone. Second, while a growing conversation remains shorter than PrefixLength, each new turn changes the hashed text and may move the session to another instance. The route stabilizes only after the request grows beyond the configured prefix length.
KV-aware routing has three components:
- Container event stream: With
SM_VLLM_KV_EVENTS_CONFIGenabled, a supported vLLM container publishes events whenever it inserts or evicts cache entries. - Managed cache index: SageMaker AI maintains an endpoint-level index from those events. It stores only one-way, truncated hashes and pointers to instances or inference-component copies; it does not retain prompts, token IDs, request payloads, or KV tensors.
- Per-request scheduling: When a request arrives, the endpoint tokenizes the prompt, finds candidates holding some or all of it, and weighs the longest cached match against current instance load. If the lookup is slow, no candidate has a useful match, or all matching instances are busy, the endpoint falls back to load-aware routing equivalent to
LEAST_OUTSTANDING_REQUESTS.
AWS has not disclosed the exact candidate-scoring coefficients. Conceptually, the utility can be approximated as U(i) ≈ αC(i) - βL(i) - γS(i), where C(i) is cached-prefix length, L(i) is instance load, and S(i) represents routing uncertainty or stale-state risk. This is an abstraction of the documented behavior, not AWS's published formula.
The index follows fleet changes. Removed instances lose their entries, while instances created by scaling or deployment updates begin with empty caches. The scheduler therefore follows evictions instead of blindly returning to a fixed hash target.
Why KV-cache reuse reduces time to first token
During prefill, an autoregressive model computes and caches the Key and Vector tensors for prompt tokens. If a later request reuses the same prefix, the runtime can read the existing KV state and run the forward computation only for new tokens. This avoids repeated attention work and intermediate-memory traffic, reducing the delay before the first generated token.
Standard dense attention has quadratic interaction scaling in sequence length. FlashAttention and related kernels dramatically improve memory movement and practical throughput, but do not remove that asymptotic compute characteristic.
Approximate KV-cache memory consumption can be expressed as:
M_KV ≈ 2 × L × H_kv × d_head × s × N
Here, L is the number of layers, H_kv the number of KV heads, d_head the head dimension, s the bytes per stored element, and N the number of cached tokens. The leading 2 accounts for Keys and Vectors. Grouped-Query Attention reduces H_kv and therefore cache pressure, but multi-turn conversations, RAG documents, and long system prompts still increase N until paging or eviction becomes necessary.
KV-aware routing does not make decoding free. It reduces the number of prompt tokens that must be prefetched or recomputed; output tokens still have to be decoded one by one, and routing does not alter model quality. The feature is most valuable when a large prefix is reused frequently and caches are under pressure. With short prompts, the saved prefill may be too small to justify tokenization and index-lookup overhead.
Performance: interpreting the reported results
AWS tested Llama 3.1 70B Instruct on seven ml.p5.48xlarge instances for 30 minutes, using 20 or 100 simulated users and prompts of up to approximately 64K tokens across multi-turn chats and document Q&A. Compared with random routing, KV-aware routing reduced p50 time to first token by as much as 97%, improved throughput by as much as 234%, and raised the KV-cache hit rate from 3%—48% to 80%—93% under long-prompt, high-load conditions.
A second, reproducible comparison used two ml.g5.xlarge instances serving Qwen3-4B-Instruct-2507. Thirty-two simulated users asked four questions each about 12 documents of roughly 7,400 tokens, resending the full conversation after every turn. Across two pairs with reversed execution order:
- CloudWatch
ModelLatencyp50 fell from 3,153/4,083 ms under random routing to 656/700 ms with KV-aware routing. - TTFT p50 for questions 2—4 fell from 1,284/1,632 ms to 237/245 ms.
- The share of prompt tokens served from cache rose from 76.1%/70.1% to 90.3%/90.3%.
- Average
OverheadLatencyincreased from 13.1/8.1 ms to 27.1/27.5 ms, adding roughly 14—19 ms.
These results show that cache-aware placement can materially improve repeated long-context workloads. They do not establish a universal improvement. The 97% TTFT reduction is an upper bound from a specific long-prompt, high-load test against random routing. The published material does not provide a direct KV_AWARE versus PREFIX_AWARE result, confidence intervals, or variance across repeated trials. The smaller experiment covers one model, one instance type, and synthetic document traffic; CloudWatch ModelLatency and client-observed TTFT are also different metrics. Production acceptance should therefore track p95/p99 latency, successful-response throughput, cache-hit rate, routing overhead, GPU memory, and cost per successful response rather than relying on average latency alone.
Engineering trade-offs and limitations
- Container dependency: At launch, KV-aware routing supports the AWS vLLM Deep Learning Container for SageMaker AI, including vLLM 0.33.0 or later and the
server-sagemaker-cuda-v2.8.0Amazon Linux build or later. It is not a portable routing protocol for arbitrary vLLM, SGLang, or custom inference stacks. - Request restrictions: It accepts OpenAI-compatible Completions and Chat Completions payloads.
InvokeEndpointWithBidirectionalStreamis unsupported. - Different isolation semantics: The feature does not provide the multi-tenant prefix isolation from the earlier prefix-aware design. It ignores
prompt_cache_keyand the prefix-aware ID header, while a non-null top-levelcache_saltcauses a 400 response. It should not be treated as a tenant-isolation boundary for sensitive workloads. - Limited decision observability: At launch, SageMaker AI exposes no dedicated KV-aware routing metric and does not report fallback events in the console or metrics. Model containers also do not deliver routing-decision logs to customer accounts. Teams must infer behavior from cached-token counts, cache-hit rate, TTFT, and
OverheadLatency. - Cold caches after scaling: New instances have no historical KV state, so TTFT can temporarily rise after scaling or instance replacement. Rolling and blue-green deployments reduce the impact window but cannot remove the first-request cold start.
- Managed implementation: Event formats, index consistency, and scheduler internals are not open source. AWS charges no additional routing fee, but instance, storage, and observability costs remain.
Deployment guidance for developers
- Check prerequisites: Use a supported vLLM GPU container with prefix caching enabled. A single-model endpoint needs at least two instances; an inference-component endpoint needs at least two copies, which may reside on one instance. A single destination accepts the configuration but provides no routing benefit.
- Enable cache events: Set
SM_VLLM_KV_EVENTS_CONFIGin the model environment to a configuration such as{"enable_kv_cache_events":true,"publisher":"zmq","endpoint":"tcp://*:5557","topic":"kv-events","replay_endpoint":"tcp://*:5658"}. These ports are for communication inside the instance and should not be exposed externally. Configure the production variant with{"RoutingConfig":{"RoutingStrategy":"KV_AWARE"}}. CUDA 13 images also require a compatibleInferenceAmiVersion. - Instrument the effect: Set
SM_VLLM_ENABLE_PROMPT_TOKENS_DETAILS=trueand inspectusage.prompt_tokens_details.cached_tokensin each response. Enable Detailed Observability to track KV-cache hit rate and TTFT through CloudWatch or PromQL. Detailed observability requires AWS CLI 2.35.6+ or Boto3 1.43.31+. - Run a three-way test: Keep the model, image, instance type, instance count, prompt set, and concurrency identical across
RANDOM,PREFIX_AWARE, andKV_AWAREendpoints. Record TTFT p50/p90/p99, end-to-end p99, cached-token ratio, effective throughput,OverheadLatency, GPU memory, and cost per successful response. - Roll out progressively: Reproduce production-style multi-turn and RAG traffic on a small endpoint, then update production endpoints with rolling or blue-green deployments. Reserve capacity for cache rebuilds and add external alerts for event-path failures and routing fallbacks.
- Match the strategy to the workload: Prefer the simpler
PREFIX_AWAREstrategy for stable shared prefixes, short prompts, moderate load, or unsupported containers. TestKV_AWAREfirst for long conversations, shared documents, high concurrency, and visible GPU-memory pressure. Use random or least-outstanding-request routing when prompts are short and interchangeable or the model is not an LLM.
Inference-component endpoints also support dynamic LoRA adapters. In that configuration, KV-aware routing only selects among instances that already have the required adapter loaded. The feature turns cache placement into a managed scheduling capability, but its economic value depends on whether the reusable-token fraction and eviction frequency outweigh tokenization, index lookup, and operational complexity.
— END —Source: AWS Machine Learning Blog (2026-11-17)
Tags: #Amazon SageMaker AI #KV Cache #Inference Routing #vLLM #LLM Serving
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments