AWS SageMaker AI Adds Health-Aware Cross-Region Inference Routing
Summary:AWS has introduced cross-Region inference routing for Amazon SageMaker AI. A Regional inference router now provides a single invocation path across real-time endpoints, inference components, and asynchronous endpoints located in different AWS Regions and accounts. The router applies local-first, health-aware traffic placement and shifts new requests when local capacity becomes unreliable. It retries only failures that are known to have occurred before model execution, while adding cross-account
What AWS Shipped
Amazon SageMaker AI now provides a Regional inference-router resource. Instead of binding an application directly to InvokeEndpoint, the client calls InvokeInferenceRouter, and the router selects among registered endpoints, Inference Components, and asynchronous endpoints that are authorized and eligible to receive the request. Existing models, containers, instance pools, and autoscaling configurations do not need to move, while direct endpoint invocations remain available.
The main abstractions are:
RoutingTarget: a logical model with one or more execution destinations that fail over independently.Destination: an endpoint or inference component identified by a full ARN, allowing the router to resolve its Region and AWS account.Local Destination: a destination in the router's own Region, which receives traffic by default.RoutingReason: a per-response explanation of why the selected destination served the request.
A router supports up to 20 routing targets, with up to 10 destinations per target. Real-time targets can mix endpoints and inference components, while an asynchronous target may contain only asynchronous endpoints.
Health-Aware Routing Loop
The router considers only registered destinations that it is authorized to invoke. When several local destinations are healthy, it distributes requests evenly among them. Remote traffic is used only when local destinations cannot serve reliably, and is allocated across healthy remote destinations in proportion to their health. Users cannot currently define weights or a fixed fallback order.
The health score is calculated from recent requests measured in the destination's own Region. Server errors, throttles, and timeouts reduce the score, with failures in the SageMaker AI platform itself having the strongest effect. Request-level errors such as invalid input do not reduce health because they are unlikely to succeed in another Region.
One important limitation is that model latency is not a health signal. An endpoint that continues returning successful responses can therefore retain traffic even if it has become substantially slower. This matters for long-form generation, batching changes, and other workloads where latency degradation may be as serious as explicit failures.
Destinations are exposed in three states:
healthy: receives normal traffic.overloaded: continues serving, but the router shifts part of its traffic away.degraded: is failing or unable to serve; the router moves most traffic away while retaining a small share for recovery detection.
Failed instance launches or an instance count below the autoscaling target can accelerate the router's response, but traffic only moves after request failures begin. Sustained errors typically trigger a shift within about a minute, while low-traffic destinations may take longer. Traffic returns over several minutes after recovery, reducing oscillation. If remote health telemetry becomes stale, the router stays in its local Region rather than moving traffic solely because monitoring data is missing.
Retry Semantics and Duplicate-Execution Risk
For a failed real-time request, the router retries once on another destination only when it can establish that the model did not run. Examples include throttling, a 503 response, a model that is not ready, rejection by a cross-Region traffic cap, or access denial by the destination. It does not automatically retry timeouts, 500 errors, or model errors because the model may already have executed even though its response was lost.
This is a deliberate boundary between failover availability and duplicate side effects, not a global exactly-once guarantee. Agent requests that invoke tools, modify databases, or trigger financial operations still need application-level idempotency keys and explicit handling of indeterminate outcomes. For asynchronous requests, the router tries eligible destinations once each, local destinations first, until one accepts the job.
Cross-Account Security and Traffic Caps
A cross-account destination requires authorization on both sides. The endpoint owner grants the router account access through a resource policy and can restrict it to one named router with sagemaker:SourceInferenceRouterArn. The router account must also have permission to invoke each destination. The router acts on the caller's behalf but does not expose endpoint access that the caller did not already possess. Status reporting requires separate DescribeEndpoint or DescribeInferenceComponent permissions.
Receiving endpoints can cap how much cross-Region traffic they accept:
C_cross = R_instance × N_in_service × p
R_instance is the measured invocation rate of one instance, N_in_service is the number of active instances, and p is the permitted cross-Region share. With 50 RPS per instance, four active instances, and a 20% cap, the endpoint accepts up to 40 cross-Region RPS. Scaling to six instances raises the cap to 60 RPS. Local-Region requests do not count against this cap, and all Inference Components on the same endpoint share it.
Engineering Trade-offs
The feature moves failure detection, retry classification, gradual traffic shifting, recovery damping, and cross-account invocation out of every application's request path. Endpoints can remain directly callable, allowing incremental migration to a unified router. The routing capability has no additional charge, but endpoints, Inference Components, CloudWatch metrics, and the always-running cost of standby capacity remain billable.
Cross-Region failover adds a network hop and router overhead, which can materially increase tail latency. AWS exposes overhead.duration but publishes no representative values. The router is itself Regional, so clients need a second router in another Region and an application-level switching strategy. It does not deploy models, copy artifacts, scale endpoints, or pre-warm remote capacity. A standby with zero active copies may fail requests until autoscaling provisions usable capacity.
Other constraints include no response streaming, an approximately one-minute real-time request timeout, a 6 MB real-time payload limit, and no IPv6 or dual-stack support for the PrivateLink interface endpoint. Because routing can send data to any registered Region, geographically constrained workloads should use separate routers and destination sets for each data-residency boundary.
What the Official Test Establishes
AWS deployed Qwen3-4B with a vLLM Inference Component in US East (Ohio) and US West (Oregon). While the local component was running, responses reported local-preferred. Removing the local component moved traffic to the remote Region with only-healthy; restoring it eventually brought traffic back to the local destination. The test also exercised a cross-account destination that lost and regained access.
This validates the control-plane state machine, destination registration, and failover path. It is not a performance benchmark. AWS does not disclose p50, p95, or p99 latency, throughput limits, detection-time distributions, cross-Region bandwidth cost, or behavior under real GPU capacity shortages. Production evaluation therefore needs fault injection using the actual model, payload, instance mix, and quota constraints.
Production Deployment Guidance
- Design two failure domains: configure two Regional routers on the client, with each router registering local and remote destinations, so the router's own Region is not a single point of failure.
- Keep the standby warm: run at least one usable copy in the secondary Region and verify that autoscaling can provision capacity within the required failover window. Capacity-aware instance pools can absorb substitution inside a Region before cross-Region routing is invoked.
- Measure real quotas: derive per-instance RPS from actual payloads, sequence lengths, and Adapter mixes before setting traffic caps and local autoscaling headroom.
- Keep client retries conservative: do not blindly retry timeouts, 500 errors, or model errors, and enforce idempotency for requests with external side effects.
- Instrument every route: log
InvokedRoutedResourceArnandRoutingReason, then alert on unhealthy destinations, error rates, and router overhead. - Separate data domains: use independent routers and destinations for workloads with different residency or compliance constraints.
- Run failure-injection tests: simulate throttling, endpoint deletion, permission revocation, failed instance launches, and network degradation to measure failover time, duplicate-execution risk, and recovery behavior.
— END —Source: AWS Machine Learning Blog (2026-11-19)
Tags: #AWS #Amazon SageMaker AI #Cross-Region Inference Routing #GPU Capacity Resilience #Distributed Inference
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments