ZICQ
中 Log in / Sign up
Newsroom Compute & Architecture #Amazon SageMaker AI #HyperPod #NVIDIA Dynamo #KV Cache #GPU Partitioning

AWS Expands SageMaker Inference Stack Across 13 Production Capabilities

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:Amazon SageMaker AI's second 2026 inference round brings together 13 launches spanning configuration benchmarking, LoRA concurrency analysis, NVIDIA MIG partitioning, managed multi-node serving with Dynamo, KV-cache-aware and cross-Region routing, continuous performance monitoring, and capacity-efficient rolling updates. The significance is operational rather than model-centric: AWS is turning inference optimization into a continuous, SLA-driven lifecycle instead of a one-time infrastructure de


Amazon SageMaker AI did not introduce a new foundation model in this 2026 round; it focused on the engineering required after a model enters production: selecting instances and containers, packing LoRA adapters and small models onto shared GPUs, serving models too large for one accelerator, routing requests according to cache state, and reducing the temporary GPU capacity required for rolling updates. The 13 launches span configuration, resource allocation, scheduling, routing, and operations.

Launch map: from one-time deployment to continuous optimization

| Stage | Capabilities | What changed | | --- | --- | --- | | Configuration and benchmarking | SageMaker AI Studio, Python SDK v3, LoRA support, SLA-based concurrency search | Uses NVIDIA AIPerf on real GPUs to rank instance, container, and optimization choices | | Multi-model deployment | Inference Components console, NVIDIA MIG | Independently scales multiple models on one endpoint and partitions a physical GPU into isolated MIG instances | | Very large model serving | Managed NVIDIA Dynamo on HyperPod | Multi-node sharding, prefill/decode disaggregation, KV-aware scheduling, and node recovery | | Traffic routing | Endpoint KV-cache awareness, Inference Gateway, cross-Region routing | Selects workers using actual cache state, queue pressure, adapter residency, and regional capacity | | Post-launch operations | Continuous performance monitoring, in-place rolling updates | Detects latency drift and idle capacity while reducing the number of newly acquired instances required for updates |

The common idea is to turn inference optimization into a repeatable engineering loop: benchmark, generate a recommendation, deploy, observe, benchmark again, and update the production fleet.

Technical mechanics

1. From manual selection to SLA-constrained search

Inference Recommendations uses NVIDIA AIPerf on real GPU infrastructure to test candidate instances, containers, and optimization techniques. Results are ranked by time to first token (TTFT), inter-token latency, throughput, and cost. Depending on the objective, the workflow can apply speculative decoding for throughput or kernel tuning for latency.

Recommendations are now available in SageMaker AI Studio, SageMaker Python SDK v3.17.0 and later, and MCP-compatible coding agents. The SDK can run recommendation jobs in parallel, resume a completed job, and connect benchmarking, configuration selection, and real-time endpoint deployment to notebooks or CI/CD pipelines.

SLA-based concurrency search asks a more operational question: how much traffic can an endpoint sustain? Its objective can be expressed as C* = max {C | TTFT(C), per-token latency(C), end-to-end latency(C), and error rate(C) all satisfy their targets}. The search begins with a broad range, probes candidate concurrency levels, and narrows the interval according to which probes pass or fail.

This is closer to production behavior than a static instance recommendation, but only if the benchmark traffic, model version, quantization, and request-length distribution are representative. Speculative decoding also depends on draft acceptance rates, while kernel tuning remains tightly coupled to the GPU model, inference engine, and model architecture.

2. LoRA and MIG reduce small-model fragmentation

Inference Recommendations now accounts for LoRA adapters. Each adapter and its associated runtime state consumes GPU memory, so adding adapters can reduce the number of concurrent requests an endpoint can sustain. AWS recommends a base-model Inference Component plus one component for each adapter, allowing adapters to be added, updated, or removed without restarting the base model.

Inference Components also allow several models to share one endpoint while each receives its own accelerator, CPU, memory, copy count, and scaling policy. A component can scale to zero when idle, but that strategy must be evaluated against cold-start latency, adapter reload time, and burst traffic rather than steady-state GPU utilization alone.

NVIDIA MIG divides a supported physical GPU into as many as seven partitions with dedicated GPU compute and memory. A model component can be assigned to a selected partition. The isolation ends at the accelerator: host CPU, host memory, networking, storage, and endpoint routing remain shared. Effective capacity therefore is not simply the sum of each model's memory requirement; host memory bandwidth, PCIe behavior, network I/O, and routing overhead must also be tested. AWS still bills the full instance-hour, so the saving comes from placing more models on one instance rather than reducing the GPU price.

3. Managed Dynamo extends the large-model boundary

For models whose weights exceed the memory of a single instance but can be sharded across nodes, HyperPod Inference now offers managed NVIDIA Dynamo. Dynamo handles model sharding, prefill/decode separation, and request placement. HyperPod supplies GPU capacity, Elastic Fabric Adapter networking, node replacement, endpoint registration, TLS, autoscaling, and observability. The Inference Operator add-on manages Dynamo, serving images, and the Grove and KAI schedulers as one deployment path.

Prefill is generally compute-intensive, while decode is more constrained by memory bandwidth and sequential generation latency. Separate GPU pools for the two phases prevent a long prompt from blocking token generation for other users. KV caches are transferred over EFA to avoid recomputing the prompt, and the service supports vLLM, SGLang, and TensorRT-LLM through an OpenAI-compatible endpoint.

The tradeoffs are equally important. Disaggregated prefill and decode add KV-transfer overhead, network contention, and scheduling complexity, while a sharded replica depends on inter-node communication. The managed integration reduces image maintenance and node-recovery work, but it also binds deployment to HyperPod Regions, EFA availability, and the AWS operating model. AWS has not disclosed absolute cross-node throughput, KV-transfer cost, or recovery-time data.

4. Three routing layers address different mismatches

Endpoint-level KV-cache-aware routing relies on cache creation and eviction events published by the model container. A central index then routes each request to an eligible instance holding the longest matching cached prefix. This is more accurate than prefix-aware routing for long prompts, multi-turn conversations, and frequently changing prefixes, but the first release requires an AWS-verified container implementing cache-event, tokenization, metrics, and OpenAI-compatible interfaces.

The HyperPod Inference Gateway solves a cluster-level problem. It exposes one OpenAI-compatible endpoint for all models in a cluster and considers KV-cache utilization, queue depth, running requests, prefix-cache hit rate, and LoRA-adapter residency. This lets it avoid workers with nearly full caches and prefer workers that already hold the requested adapter.

Cross-Region routing addresses capacity at the outermost layer. The application invokes a local endpoint first and can fail over to compatible endpoints in another Region or AWS account. Traffic uses AWS PrivateLink, while routing policies can cap fallback volume and restrict the Regions allowed to serve a request. The router does not replicate models, deploy endpoints, or create capacity. Cross-Region fallback therefore adds network latency and data-residency obligations while depending on the availability of independently deployed capacity.

5. Monitoring and in-place updates complete the lifecycle

SageMaker AI can now continuously monitor latency, GPU utilization, and serving configuration on generative AI inference endpoints. It identifies latency drift after traffic changes, configuration gaps that existed since deployment, and GPU capacity that remains idle. The service does not inspect request or response content, so operational drift detection cannot detect semantic-quality or safety regressions. Week-over-week comparisons may also misclassify seasonal traffic as a regression.

In-place rolling updates reduce the need to acquire new capacity for every batch. The system provisions one small temporary batch, shifts traffic to it, and then updates and reuses the instances released by that batch. In AWS's example, a 20-instance fleet with a batch size of two requires two additional instances rather than 20. The remaining costs are temporary capacity during the rollout and a short period in which different model versions coexist. Traffic draining, failure recovery, and version compatibility still need production validation.

Measured results, benchmark quality, and engineering costs

AWS reports that the Inference Gateway reduces P99 TTFT by up to 97% versus round-robin routing on mixed GPU generations and by up to 98% under bursty traffic. This is the strongest performance claim in the release, but it is still a vendor-reported relative result. The source provides no absolute TTFT, request distribution, concurrency level, cache-hit rate, sample size, or variance, so it should not be generalized to other models and deployments.

Running AIPerf on real GPUs is more credible than a paper-only benchmark, but a fair comparison must hold the model and quantization version, input/output-token distribution, batch size, warm-up, LoRA count, serving-engine version, concurrency ladder, and error threshold constant. Results should include P50/P99 TTFT, inter-token latency, throughput, cache-hit rate, and cost per million tokens rather than only percentage improvements.

| Capability | Main benefit | Engineering cost to include | Evidence boundary | | --- | --- | --- | --- | | Recommendations and SLA search | Selects instances and maximum feasible concurrency | Benchmark time, candidate count, workload representativeness | No unified absolute benchmark published | | LoRA Components | Independently manages adapters and scaling | Adapter residency, routing hits, cold starts | Concurrency must be retested at each adapter count | | MIG | Increases density for small models sharing a GPU | Host contention and full-instance billing | Up to seven partitions, depending on GPU | | Managed Dynamo | Serves models that do not fit one instance | Multi-node communication, KV transfer, platform dependency | No absolute cross-node performance data | | KV-aware routing | Improves reuse of actual cached prefixes | Event reporting, index consistency, container certification | Hit rate and tail latency still need measurement | | Gateway | Routes according to live GPU state | Metadata overhead and load skew | Only relative P99 TTFT reductions of up to 97%/98% | | Cross-Region routing | Uses idle regional capacity and provides fallback | Cross-Region latency, residency, backup capacity | The router does not add total capacity | | In-place updates | Reduces new capacity needed for large rollouts | Temporary batch and mixed-version window | Example avoids 18 extra instances for a 20-instance fleet |

Developer deployment guidance

  1. Create a reproducible baseline. Record TTFT, inter-token latency, end-to-end latency, P95/P99, throughput, error rate, and cost per million tokens using representative request samples. Preserve the model, container, GPU, and quantization configuration with every result.
  2. Integrate recommendations into CI/CD. Use Python SDK v3 to compare serving frameworks such as LMI and vLLM in parallel. Rerun recommendations whenever traffic, model versions, or adapter counts change instead of retaining a static launch-day configuration.
  3. Use SLA search to set concurrency limits. Apply TTFT, per-token latency, error-rate, and cost constraints together, and record feasible_max. Average latency alone is insufficient for capacity planning.
  4. Evaluate Inference Components and MIG first for small models. Confirm that each model fits its intended partition, then test shared CPU, memory, and network pressure. Include cold-start effects when burst traffic is likely to trigger scale-from-zero.
  5. Move to HyperPod and Dynamo only when single-instance limits require it. Measure cross-node sharding, prefill/decode allocation, KV-transfer overhead, node-replacement time, and end-to-end cost. Merely fitting the weights is not an operational acceptance test.
  6. Choose routing according to cache observability. Use prefix-aware routing when container changes are impractical; enable event-driven KV-cache routing when a verified container is available and prefixes change quickly. Validate cluster tail latency and cross-Region failover delay separately.
  7. Keep human gates in the operational loop. Drift Watch and the aws-ai-ml skill can generate findings and SDK code, but deployments should retain review, least-privilege permissions, and rollback procedures. Operational monitoring does not replace independent model-quality evaluation.

The value of these 13 launches is not that every inference request becomes faster. It is that SageMaker can now execute a measurable SLA-and-cost loop spanning benchmarking, resource packing, cache-aware scheduling, regional failover, and rolling deployment, reducing the amount of custom platform code that production teams must maintain.


Source: AWS Machine Learning Blog (2026-11-20)

— END —

Tags: #Amazon SageMaker AI #HyperPod #NVIDIA Dynamo #KV Cache #GPU Partitioning

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…