Amazon Web Services made cross-Region inference routing in Amazon SageMaker AI generally available on November 19, 2026. A new resource, the inference router, gives an application one invocation path across SageMaker endpoints and inference components in one or more Regions and accounts. According to the AWS Machine Learning Blog post, the router "prefers its own Region and shifts new requests to another destination only when the local one can't serve reliably," and returns traffic gradually after recovery.
This matters because GPU capacity is the scarce part of serving models. If your instance type runs out in one Region, requests throttle even when the same model has spare capacity in another. Until now, SageMaker users had to build the failover logic themselves. This post explains how the router decides, what it costs, how to set it up and where it stops.
TL;DR: SageMaker cross-Region inference routing
| Question | Answer |
|---|---|
| What is new? | An inference router resource that fails over across Regions and accounts |
| Status | Generally available, most commercial Regions except me-south-1 and me-central-1 |
| Works for | Real-time and asynchronous inference |
| Cost | No extra charge; pay for endpoints and CloudWatch metrics |
| Streaming? | Not supported |
| Does it add capacity? | No; destinations must already be running |
| Limits | Up to 20 routing targets per router, 10 destinations per target, 6 MB real-time payloads |
| Retries | Real-time requests retried once, only when the model did not run |
The problem it solves
A SageMaker endpoint lives in one Region and one account. AWS describes the failure mode plainly: when autoscaling cannot add instances of the type your endpoint needs, requests throttle and time out, even when the same model has spare capacity in another Region.
To use that spare capacity yourself, the post says, your application must detect a struggling endpoint, decide which errors are safe to retry without running the model twice, move traffic back after recovery without flapping, and manage access to endpoints in other accounts. That code sits on the critical path of every request, and every team that serves across Regions maintains its own version.
The router moves that work into the service. You register destinations, and the application invokes the router instead of a specific endpoint. If you already run capacity-aware serving, see our earlier notes on SageMaker KV-cache aware routing for vLLM endpoints and on MIG multi-instance GPU endpoints, which tackle utilization inside a Region rather than across Regions.
The vocabulary
| Term | Meaning |
|---|---|
| Inference router | A Regional SageMaker resource in your account that your app invokes |
| Routing target | A name for one logical model on a router, such as fraud-model; each fails over independently |
| Destination | A real-time endpoint, inference component or asynchronous endpoint, identified by ARN, which encodes its Region and account |
| Local destination | A destination in the router's own Region, preferred while healthy |
| Routing reason | A value returned with each real-time response explaining where the request went |
A routing target holds either real-time destinations, which can mix endpoints and inference components across Regions and accounts, or asynchronous endpoints only. The Region where you create the router is its local Region; to prefer a different Region, create a router there.
How the router decides where a request goes
The logic, as AWS describes it:
- The router considers only destinations registered on the routing target that it is authorized to invoke.
- It prefers destinations in its own Region and spreads requests evenly across several healthy local ones.
- Requests leave the Region only when local destinations cannot serve reliably.
- SageMaker computes a health score per destination from recent requests, measured in the destination's own Region. Server errors (5xx), throttles and timeouts lower the score. Errors from the SageMaker platform, such as a failed health check, lower it the most. Errors caused by the request itself, such as invalid input, do not.
- Model latency is not an input, so an endpoint that keeps returning successful responses keeps its traffic.
Health is published to CloudWatch as one of three states: healthy (serves normally), overloaded (still serves, part of its traffic moves away) and degraded (failing, most traffic moves away, with a small share kept to detect recovery).
You do not tune thresholds. AWS manages them. If health information for remote destinations goes stale, the router keeps traffic in its own Region, so a monitoring gap never moves traffic on its own.
When traffic moves and when it returns
The more requests fail, the more traffic moves, so a destination need not fail outright first. Per the post, traffic starts to move within about a minute of sustained errors, or longer for a destination with little traffic. Traffic leaving the local Region is split across healthy remote destinations in proportion to their health, and you cannot set weights or a fallback order. After recovery, traffic returns over several minutes, more slowly than it left. Requests in progress are not interrupted.
Retries
When a destination fails a real-time request without running the model, for example a throttle, a 503, a model that is not ready, a rejection by a cross-Region traffic cap or access denied, the router retries once on a different destination. It never retries timeouts, 500 errors or model errors, because the model may already have run. For asynchronous requests it tries each eligible destination once, local first, until one accepts.
Setup in outline
AWS's example serves Qwen3-4B with the vLLM container as an inference component in US East (Ohio) and US West (Oregon). You create a router in the first Region with one routing target that lists both destinations by ARN, then confirm each destination's RoutingStatus is Routable with DescribeInferenceRouter. The application calls the router with the routing target name and the same payload it would send to InvokeEndpoint, and reads InvokedRoutedResourceArn and RoutingReason from the response.
In AWS's test, traffic stayed in us-east-2 with reason local-preferred while the local component ran. After the team removed that component, requests moved to us-west-2 with reason only-healthy, with no change to the application. After restoring it, traffic came back to us-east-2. A complete notebook is on GitHub, per the post.
The four routing reasons are local-preferred, capacity-available, failover and only-healthy. AWS recommends logging both the destination ARN and the reason with your own request logs.
Cross-account routing and permissions
A central platform account can own the router while other accounts own the endpoints. When the destination is in a different account:
- The endpoint owner attaches a resource-based policy that allows the router account to invoke the endpoint. The
sagemaker:SourceInferenceRouterArncondition key limits that access to one named router. - A second statement lets the router account call
DescribeEndpoint; without it the destination shows an unknown status. - Callers in the router account need
sagemaker:InvokeInferenceRouteron the router andsagemaker:InvokeEndpointon every destination, because the router invokes each destination on the caller's behalf.
AWS stresses that a router never gives a caller access to an endpoint they could not call directly. For the identity-scoping principle behind that, compare our post on Bedrock Knowledge Base ACLs on AgentCore Gateway.
The cross-Region traffic cap
Failover has a cost for the receiving Region: overflow lands on endpoints that already serve local callers. By default a receiving endpoint accepts any amount of cross-Region traffic, so a surge can push it into throttling.
The owner can set CrossRegionTrafficConfig in CreateEndpointConfig with two values:
MaxInvocationsPerSecondPerInstance: what one instance can take, measured by a load test of your model, instance type and payload.MaxCrossRegionPercentPerEndpoint: the share of capacity other Regions may use.
SageMaker multiplies the rate by instances in service and applies the percentage, so the cap follows autoscaling. AWS's example: 50 invocations per second per instance, a 20 percent cap and 4 instances gives 40 requests per second from other Regions; with 6 instances, 60. Only requests from other Regions count. Local calls are never throttled by the cap. Over-cap requests are rejected before the model runs, and the router retries each once elsewhere. If no destination can take it, the app gets a throttling error. The cap applies to real-time endpoints, and all inference components on an endpoint share it.
Monitoring
Per request, the response carries the serving destination and the routing reason. Asynchronous responses do not include the destination; use CloudWatch metrics to see placement. The router publishes OpenTelemetry metrics to CloudWatch, including invocations per destination, destination duration, router overhead duration, and destination status (healthy, overloaded or degraded). AWS suggests alarming on destinations that are not healthy and on overall error rate. CloudTrail records router API calls as management events and invocations as data events you can enable. Our coverage of SageMaker HyperPod enhanced observability shows the same direction on the training side.
Limits and planning notes
- No streaming. If your app streams tokens, the router is not a fit yet.
- About a minute. A real-time request not answered in about a minute fails with a service-unavailable error. Do not retry that automatically: the model may have run.
- Quotas. Up to 20 routing targets per router, 10 destinations per target, 6 MB real-time payloads.
- Capacity is yours. The router does not scale endpoints or copy models. A remote inference component with zero copies gets a small share of failover traffic and those requests fail until copies exist. Keep at least one running, or confirm autoscaling scales out.
- The router is regional. If its Region is impaired, calls fail. AWS suggests routers in two Regions and client-side switching.
- Data residency. A request may be processed in any Region where you registered a destination. Use separate routers per geography.
- Clean up. The router never removes a deleted destination on its own.
- Private access. Apps in a VPC can use an interface VPC endpoint (PrivateLink), IPv4 only.
What it costs
AWS states there is no additional charge for routing. You pay standard SageMaker AI pricing for the endpoints and inference components that serve requests, whether directly or through the router, and standard CloudWatch rates for router metrics. See SageMaker pricing for rates. The real cost is standby capacity: a second Region only helps if it has instances ready.
Who should adopt it
| If you | Then |
|---|---|
| Serve a latency-sensitive model on scarce GPU types in one Region | Consider a second Region behind a router |
| Stream tokens to users | Wait; streaming is unsupported |
| Run a platform team serving many models across accounts | One router with a routing target per model fits |
| Have strict data residency | Use one router per geography |
Bottom line
The router turns a pile of custom failover code into a managed resource, with health-based shifting, a single retry rule and a cap that protects the receiving side. It does not make GPUs appear. Treat it as insurance for capacity shortfalls, and budget for the standby capacity that makes the insurance work.
Related reading
- SageMaker KV-cache aware routing for vLLM endpoints
- SageMaker inference with NVIDIA MIG multi-instance GPUs
- SageMaker HyperPod enhanced observability
- Bedrock Knowledge Base ACLs on AgentCore Gateway
- SageMaker serverless model customization with FFT
- Bedrock AgentCore Identity with private CA
Details reflect the AWS announcement of November 19, 2026. Availability and quotas may change.
