AWS Adds Managed NVIDIA Dynamo to SageMaker HyperPod for Disaggregated LLM Serving
Summary:Amazon SageMaker HyperPod Inference now supports managed NVIDIA Dynamo, bringing multi-node serving, disaggregated prefill/decode, and KV-aware routing into AWS's managed control plane. The HyperPod Inference Operator, Dynamo Operator, Grove, and KAI Scheduler coordinate endpoints, gang scheduling, GPU placement, autoscaling, and recovery; teams can either bring an existing DynamoGraphDeployment or use InferenceEndpointConfig with an OpenAI-compatible endpoint. In AWS's internal test, disaggreg
What Changed: Dynamo Moves Into the Managed Control Plane
Amazon SageMaker HyperPod Inference can now use the open-source NVIDIA Dynamo through a HyperPod Inference add-on. This brings multi-node serving, disaggregated prefill/decode, and KV-aware routing into the SageMaker endpoint, scheduling, storage, and operations stack. Dynamo supplies the distributed inference data plane, while AWS adds component-version alignment, gang placement, EFA networking, node-health monitoring, and replacement. Teams can declare a topology through either InferenceEndpointConfig or an existing DynamoGraphDeployment.
The integration targets a hard capacity boundary for very large models. GLM-5.3, for example, has FP8 weights that exceed the 640 GB of GPU memory available on one ml.p5.48xlarge instance before KV Cache or runtime overhead is included. The model therefore cannot fit that instance configuration, and once execution crosses node boundaries, communication directly affects time to first token. Parallelism, scheduling, and KV movement must be designed as one system rather than independent components.
Two Deployment Paths
| Path | What the team owns | What the platform manages | Best fit |
|---|---|---|---|
| Bring-your-own DynamoGraphDeployment | Full DGD, container image, storage, networking, pod placement, endpoint, TLS, registration, and autoscaling | Dynamo Operator manages the DGD; HyperPod manages physical nodes | Existing Dynamo users needing direct control of the serving graph |
| InferenceEndpointConfig with dynamoSpec | Model declaration and Dynamo-specific settings | Weight staging from S3 or FSx, endpoint, TLS, registration, KV-routing defaults, and KEDA-based autoscaling | Existing HyperPod users seeking a managed standard endpoint |
Both paths support single-instance and multi-node deployments. The managed path defaults to vLLM while allowing SGLang or TensorRT-LLM, and exposes an OpenAI-compatible request format through SageMaker Runtime.
How the Control Plane Fits Together
The add-on installs a coordinated set of controllers rather than a standalone Dynamo container:
- The HyperPod Inference Operator manages endpoints through
InferenceEndpointConfigand translatesdynamoSpecinto Dynamo configuration. - The Dynamo Operator reconciles DynamoGraphDeployment resources, including frontends, model workers, replica counts, and resource requirements.
- Grove manages groups of related pods and their scaling relationships so a multi-node worker is not scaled or restarted in fragments.
- KAI Scheduler applies gang scheduling, placing all pods required by a multi-node worker together.
- AWS Deep Learning Containers provide the Dynamo runtime, while the add-on manages compatible component versions.
This control plane removes much of the Kubernetes, GPU scheduling, networking, and recovery assembly that a team would otherwise have to operate directly.
Prefill, Decode, and the KV Data Path
In an aggregated deployment, prefill and decode run in the same worker. Prefill processes the input prompt in parallel, produces the first token, and builds the KV Cache used by subsequent decoding. Decode then generates output token by token. Both phases share model weights and local KV Cache, eliminating an inter-worker state handoff. The tradeoff is that a long prefill phase can compete with decode for GPU compute and memory bandwidth.
In a disaggregated deployment, the frontend sends requests to separate prefill and decode pools. The prefill pool computes the prompt and produces KV state; that state crosses Elastic Fabric Adapter to the decode pool, which continues generation. The pools can use different replica counts, parallelism settings, and scaling policies. This is useful for long prompts, high concurrency, or workloads whose prefill and decode demand grow at different rates.
The cache footprint for one sequence can be approximated as:
KV_bytes ≈ 2 × L × H_kv × d_head × s × T
Here, L is the number of layers, H_kv is the number of KV heads, d_head is the head dimension, s is bytes per element, and T is the number of cached tokens. The factor of two accounts for keys and values. With multiple active sequences, the aggregate footprint depends on concurrency and output length.
Dynamo's KV-aware router considers reusable prefix KV state and live worker load. Prefix reuse can avoid repeated prefill of shared system prompts, tool definitions, or long documents, improving TTFT, while load awareness helps prevent a hot worker from becoming a bottleneck. Production deployments still need to validate cache-index consistency, eviction behavior, hash collisions, tenant isolation, and routing feedback latency.
A frequently overlooked cost is that both the prefill and decode pools load the full model weights. In the GLM-5.3 example, one prefill replica and two decode replicas create three logical copies of the model. Disaggregation improves resource allocation and scaling flexibility; it does not inherently reduce weight storage or model-loading cost.
Tensor Parallelism, Pipeline Parallelism, and Recovery
A Dynamo worker can run on one instance or span instances with tensor parallelism or pipeline parallelism. Tensor parallelism shards matrix operations across GPUs and requires frequent collective communication such as all-gather and reduce-scatter, making it sensitive to interconnect bandwidth and latency. Pipeline parallelism shards model layers, reducing some layer-boundary communication but introducing pipeline bubbles, stage imbalance, and more complex batching.
EFA networking and KAI Scheduler address complementary parts of this problem: the former supplies the high-performance fabric, while the latter ensures that all GPUs required by a worker are placed together. HyperPod monitors node, GPU, and EFA health and replaces failed nodes; orchestration components then recreate affected workers. This reduces manual recovery work, but application teams must still design safe request retries, idempotency, KV-state recovery, and traffic draining for the rebuild interval.
Engineering Trade-offs
| Dimension | Aggregated serving | Disaggregated serving | |---|---|---| | Weight memory | Prefill and decode share one weight set | Each pool holds a complete weight set, increasing replication cost | | KV transfer | No cross-pool handoff | EFA/RDMA transfer enters the latency-sensitive path | | Autoscaling | Simpler, but the two phases may interfere | Prefill and decode scale independently, adding allocation flexibility | | Scheduling | Smaller topology and failure surface | More pools and replica ratios to coordinate | | Best fit | Short prompts, little phase interference, network-sensitive workloads | Long prompts, high concurrency, or visibly imbalanced prefill/decode demand |
Teams already using Dynamo can preserve their recipes, manifests, and debugging workflows through the bring-your-own DGD path. The InferenceEndpointConfig path is easier for HyperPod users because endpoint creation, TLS, weight staging, and autoscaling are managed. In either case, Dynamo remains open source; the add-on governs integration versions and infrastructure behavior rather than locking model weights or inference engines.
What the Official Measurements Show
AWS presented two AIPerf test sets:
- Qwen3-235B-A22B-FP8 with TensorRT-LLM on 16 H200 GPUs: Requests used 4,000 input tokens and 200 output tokens. AWS retained only points with P90 TTFT below one second and median generation speed of at least 50 tokens per second per user. The best qualifying disaggregated point reached 207 output tokens per second per GPU, versus 101 for aggregated serving, a 2.05x ratio. The corresponding concurrency levels were 80 and 32.
- Qwen3.5-397B-A17B-FP8 with vLLM, TP4 × PP2, and eight H200 GPUs: This model could fit on one instance, so AWS deliberately split it across two to isolate the node-boundary effect. Under a high-prefix-reuse agent workload at concurrency 32, the two-node configuration delivered 126.4 output tokens per second per GPU and retained 99.1% of single-node throughput.
The second result indicates that, for this model, hardware, and parallelism setup, crossing the node boundary did not create a major throughput cliff. It does not prove that every model requiring multiple nodes will scale identically, because the tested model already fit within one instance.
The first result also needs careful interpretation. The 207 and 101 figures are the highest qualifying points under different concurrency, batch, and prefill/decode conditions, not a fixed-load ablation. The post does not report p99 latency, cost per million tokens, power consumption, scale-up cold starts, failure-recovery time, or independent reproduction. The 2.05x figure is therefore an attainable operating point in the vendor environment, not a universal multiplier for disaggregated serving.
Production Deployment Checklist
- Build a complete memory budget: Account for full weights in every replica, peak KV Cache at the intended context length, runtime overhead, and fragmentation. Parameter-file size alone is not a capacity plan.
- Choose the topology from workload behavior: Test aggregated serving first for short prompts or low phase interference. Introduce disaggregation when long prompts, concurrency, or distinct prefill/decode growth curves justify the extra control complexity.
- Verify the EFA path: Confirm that cross-instance collectives and prefill-to-decode KV transfers use the intended EFA/RDMA transport rather than silently falling back to a general network stack.
- Use AISimulate to narrow the search space: Compare aggregated and disaggregated modes, parallelism, and pool ratios for the selected model, engine, and GPU before consuming real GPU time.
- Replay production traffic with AIPerf: Use realistic input length, output length, and prefix-reuse distributions. Sweep concurrency and enforce separate TTFT, per-user generation-speed, and tail-latency objectives.
- Tune topology before engine flags: Settle the serving mode, parallelism, and pool allocation first, then adjust batching, quantization, and vLLM/SGLang/TensorRT-LLM settings one variable at a time.
- Exercise scaling and failure paths: Check whether S3/FSx weight loading constrains KEDA expansion, whether retries remain safe during worker replacement, and whether prefill and decode pools drift out of balance.
- Account for total cost: Include H200 capacity, cross-node communication, duplicate weight storage, cold-start caches, additional frontends, KV transfers, and recovery overhead—not just instance price.
The important change is not a new attention algorithm. It is the placement of Dynamo's distributed inference data plane inside SageMaker's managed scheduling, storage, networking, and operations control plane. That lowers the barrier to operating multi-node large-model services, but the architecture only lowers total cost when the topology matches real traffic and when cache, parallelism, and capacity decisions are validated with production distributions.
— END —Source: AWS Machine Learning Blog (2026-11-17)
Tags: #NVIDIA Dynamo #SageMaker HyperPod #Disaggregated Serving #KV-Aware Routing #Multi-Node Inference
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments