ZICQ
中 Log in / Sign up
Newsroom Compute & Architecture #AWS #Amazon SageMaker HyperPod #LLM Inference #Cross-Region Routing #GPU Scheduling

AWS Launches HyperPod Global Inference Gateway for Cross-Cluster, Cross-Region Routing

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:AWS has introduced the Amazon SageMaker HyperPod Global Inference Gateway, a managed endpoint that routes inference traffic across HyperPod clusters within one AWS Region or across multiple Regions. The global tier scores healthy clusters using KV-cache headroom, queue depth, Region proximity, and recent error rate, while the existing cluster-local Inference Gateway continues selecting the model pod. The service adds automatic cluster failover, SigV4 and IAM authorization, and PrivateLink-based


Architecture: Separating Cluster Selection from Pod Selection

The Amazon SageMaker HyperPod Global Inference Gateway extends the existing cluster-local inference entry point into a unified endpoint spanning HyperPod clusters within one AWS Region or across multiple Regions. Clients interact with one regional gateway endpoint instead of maintaining application-side mappings between models, clusters, and endpoints.

A request follows this path:

  1. The client signs the request with AWS Signature Version 4, or SigV4, and sends it to the gateway's regional endpoint.
  2. The AWS-managed routing fleet terminates TLS and validates the SigV4 signature, IAM authorization, and rate-limit status.
  3. The global gateway scores healthy clusters using their reported load signals and selects a destination.
  4. PrivateLink carries the request over the AWS backbone to the selected cluster, either within the same Region or across Regions.
  5. The cluster-local Inference Gateway selects the model pod using real-time signals such as KV-cache utilization and queue depth.

The result is a clear two-tier scheduler. The global tier allocates capacity across clusters and Regions, while the local tier performs GPU-aware pod placement. Cross-Region routing therefore does not bypass the cache- and memory-awareness already available inside each cluster.

When a cluster is published, the HyperPod Inference Controller creates a Network Load Balancer and VPC Endpoint Service in the customer's account, then uploads the listener certificate to the TLS bucket. The Global Inference Gateway validates these resources and receives continuous load reports from a service-managed metrics publisher.

Routing Mechanics: Four Signals Determine Traffic Placement

According to the documented behavior, the gateway scores every healthy cluster for each request. A conceptual model is:

S(c) = Σ w_i · z_i(c)

Here, c denotes a candidate cluster, w_i is a routing weight, and z_i(c) is the corresponding normalized signal. AWS does not disclose the normalization formula, metric sampling interval, aggregation window, or score-refresh rate, so the exact scheduling behavior cannot be inferred from the weights alone.

The default weights are:

  • KVHeadroom: 0.20 for available KV cache and GPU memory.
  • QueueDepth: 0.25 for request backlog.
  • RegionProximity: 0.40 for distance between the cluster and the gateway Region.
  • ErrorRate: 0.15 for the recent failed-request ratio.

All four weights must sum to 1. Administrators can replace the entire set but cannot adjust only one signal. Raising recent-error sensitivity to 0.7, for example, requires reducing the other three weights. Routing is supplied to CreateGlobalInferenceGateway and can later be changed with UpdateGlobalInferenceGateway. Weight changes do not reprovision capacity, while in-flight requests finish under the previous policy.

The defaults favor nearby clusters. That can reduce network latency and cross-Region transfer cost, but it may also leave requests queued on a saturated local cluster while remote GPUs remain idle. Increasing the weights for RegionProximity or KVHeadroom can improve resource utilization at the cost of longer network paths and greater data-locality pressure.

The public semantics also imply that the routing layer maintains candidate state for every healthy cluster, making scoring logically linear in the number of clusters. AWS does not state whether scores are cached, batched, or incrementally updated, nor does it publish metadata latency.

Failover and Security Boundaries

The gateway continuously health-checks registered clusters. When one fails, new requests move to the next-best cluster. A failed request is retried once on another cluster unless response streaming has already started. If no healthy cluster remains, the gateway returns 503.

That retry boundary matters in production. Non-streaming generation requests can generally move to another backend, but once a streaming response has emitted tokens, automatic replay can duplicate content, corrupt client state, or create billing discrepancies. High-concurrency services should not treat the gateway as a general idempotency layer; request IDs, session state, and client-side deduplication remain necessary.

Security is enforced through three layers:

  • Every client request must carry a valid SigV4 signature and pass IAM authorization. Streaming invocation uses a separate IAM action.
  • The gateway checks authentication, permissions, and rate limits before a request reaches any cluster.
  • Traffic between the gateway and clusters uses PrivateLink rather than a public entry point. AWS states that the gateway does not log or store prompts or completions, although application-side logging, tracing, and compliance auditing remain the customer's responsibility.

For customer identity validation, a JWT can be passed through X-Amzn-SageMaker-Customer-Identity-Token without a Bearer prefix. The token is included in the SigV4 signature; an unsigned token returns 403 and a repeated token returns 400. JWT validation should be enabled on either every cluster serving a given model or none of them, preventing inconsistent authentication outcomes after routing.

Engineering Trade-offs: Capacity, Latency, and Stability Are Not Free

Cross-Region overflow improves utilization but increases tail latency. Model-generation length, context size, and KV-cache hit rate all affect the value of remote routing. Short requests may incur noticeable network cost, while long-context workloads amplify transfer and serialization overhead.

Weighted scoring can create feedback loops and oscillation. If several clusters simultaneously report high queue depth or error rates, traffic may converge on a small set of fallback clusters. The announcement does not describe hysteresis, lag handling, outlier filtering, or maximum migration rates, so production teams should monitor whether routes move too frequently.

Automatic failover does not make every operation replay-safe. Cluster failure, interrupted streaming, and cross-Region retries can alter client-visible state. For agent tool calls or generations with external side effects, gateway-level retries cannot determine whether a tool has already executed. Idempotency keys and execution records still belong in the application layer.

Cross-Region data residency requires separate review. PrivateLink and the AWS backbone provide private connectivity, but they do not keep data inside the source Region. Regulated workloads must verify that the selected cluster's Region complies with the applicable data-residency policy.

The managed service introduces billing and platform coupling. A published cluster that is not attached to any gateway receives no traffic, but its Network Load Balancer still runs and incurs charges. Teams must also account for PrivateLink resources, gateway invocations, and cross-Region data transfer. Although the API accepts OpenAI-compatible requests, SigV4 signing requires AWS-aware client integration; this is not a drop-in replacement for every OpenAI SDK.

Data-plane routing does not automatically prove endpoint control-plane high availability. The announcement explains that requests can move across Regions and that AWS operates the routing fleet, but it does not disclose the SLA, multi-AZ redundancy, or control-plane failover behavior of the regional entry point. Production adopters must confirm the reconnect and alternate-Region strategy for an unavailable gateway endpoint.

Performance Evidence and Benchmarking Limits

Bagelcode reported that the gateway routed requests across two AWS Regions through one endpoint and continued serving when a cluster could no longer serve the model. This demonstrates that cross-Region placement and cluster-level failover reached customer beta use, but the evidence is qualitative.

The announcement provides no end-to-end latency, per-node throughput, concurrency ceiling, failure-detection time, recovery time, KV-cache utilization gain, cross-Region byte volume, cost per million tokens, or comparison against a self-managed DNS, global load balancer, and NLB stack. A stable performance improvement therefore cannot be derived from the official material.

A more reliable evaluation should keep model versions, prompts, output lengths, batch sizes, and client hardware constant while comparing direct cluster endpoints with the Global Inference Gateway. It should record TTFT, TPOT, end-to-end P95 and P99 latency, throughput, 5xx rate, recovery time, KV-cache hit rate, and cost per million tokens. Short requests, long contexts, streaming responses, and burst traffic should be evaluated separately.

Developer Deployment Recommendations

  1. Standardize model mappings first. Every cluster's InferenceGatewayConfig should include the relevant scheduler, and clusters that share traffic should use the same modelName. A model without a scheduler returns 404. Production clusters should also align model versions, quantization, and inference parameters so one model name does not conceal behavioral drift.
  2. Publish only necessary clusters. Enable spec.globalRouting in InferenceGatewayConfig, then call CreateGlobalInferenceGateway. Publication creates billable resources such as an NLB, so clusters that will not receive traffic should not be published prematurely.
  3. Group clusters by failure domain and policy. Models subject to strict data residency should use a gateway containing only local-Region clusters. Models that permit cross-Region overflow can include multiple Regions, but every fallback Region must retain enough headroom to absorb a failure.
  4. Minimize IAM and customer-identity scope. Callers should receive only InvokeGlobalInferenceGateway and InvokeGlobalInferenceGatewayWithResponseStream. The service execution role should access only registered clusters, the required EKS APIs, and TLS-bucket resources under the gig/ prefix.
  5. Keep migration and rollback paths. Retain each cluster's original Inference Gateway endpoint while clients move to the global endpoint, and switch back if necessary. Deleting the gateway leaves the per-cluster endpoints operational, providing a rollback route.
  6. Run explicit failure drills. Test complete cluster loss, error-rate spikes, single-Region disconnection, interruption after streaming begins, routing-weight changes, and the no-healthy-cluster path. Verify client retry, deduplication, and 503 handling.
  7. Tune weights against real SLOs. Latency-sensitive services can increase RegionProximity; elasticity-oriented services can increase KVHeadroom; resilience-oriented scenarios can increase ErrorRate. Monitor route distribution after every change to avoid traffic moving repeatedly between clusters.
  8. Calculate the full cost. Review NLB, VPC endpoint, gateway invocation, and cross-Region data-transfer charges in addition to GPU instances, and remove clusters that are published but no longer carry traffic.

The Global Inference Gateway is available in 13 AWS Regions. It is most appropriate for teams already operating HyperPod with multiple model replicas and a requirement for cross-cluster or cross-Region availability. For a single Region and a single cluster, the existing cluster-level Inference Gateway remains the simpler option.


Source: AWS Machine Learning Blog (2026-11-20)

— END —

Tags: #AWS #Amazon SageMaker HyperPod #LLM Inference #Cross-Region Routing #GPU Scheduling

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…