explainx.ai350K+ learn from us
TrendingAI News TodayExplainx for Business
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • TL;DR: SageMaker cross-Region inference routing
  • The problem it solves
  • The vocabulary
  • How the router decides where a request goes
  • Setup in outline
  • Cross-account routing and permissions
  • The cross-Region traffic cap
  • Monitoring
  • Limits and planning notes
  • What it costs
  • Who should adopt it
  • Bottom line
  • Related reading
← Back to blog

explainx / blog

SageMaker Cross-Region Inference Routing: Failover Without Writing Your Own Router

Amazon SageMaker, AWS, AI Infrastructure, Model Serving

Part of Microsoft, Apple and Amazon AI

Amazon SageMaker AI adds cross-Region inference routing: one router, health-based failover, a traffic cap, no extra charge. How it works, limits and setup.

Nov 19, 2026·10 min read·Yash Thakker
add explainx.ai
go deep
SageMaker Cross-Region Inference Routing: Failover Without Writing Your Own Router

Amazon Web Services made cross-Region inference routing in Amazon SageMaker AI generally available on November 19, 2026. A new resource, the inference router, gives an application one invocation path across SageMaker endpoints and inference components in one or more Regions and accounts. According to the AWS Machine Learning Blog post, the router "prefers its own Region and shifts new requests to another destination only when the local one can't serve reliably," and returns traffic gradually after recovery.

This matters because GPU capacity is the scarce part of serving models. If your instance type runs out in one Region, requests throttle even when the same model has spare capacity in another. Until now, SageMaker users had to build the failover logic themselves. This post explains how the router decides, what it costs, how to set it up and where it stops.

TL;DR: SageMaker cross-Region inference routing

table · 2 cols
QuestionAnswer
What is new?An inference router resource that fails over across Regions and accounts
StatusGenerally available, most commercial Regions except me-south-1 and me-central-1
Works forReal-time and asynchronous inference
CostNo extra charge; pay for endpoints and CloudWatch metrics
Streaming?Not supported
Does it add capacity?No; destinations must already be running
LimitsUp to 20 routing targets per router, 10 destinations per target, 6 MB real-time payloads
RetriesReal-time requests retried once, only when the model did not run
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

The problem it solves

A SageMaker endpoint lives in one Region and one account. AWS describes the failure mode plainly: when autoscaling cannot add instances of the type your endpoint needs, requests throttle and time out, even when the same model has spare capacity in another Region.

To use that spare capacity yourself, the post says, your application must detect a struggling endpoint, decide which errors are safe to retry without running the model twice, move traffic back after recovery without flapping, and manage access to endpoints in other accounts. That code sits on the critical path of every request, and every team that serves across Regions maintains its own version.

The router moves that work into the service. You register destinations, and the application invokes the router instead of a specific endpoint. If you already run capacity-aware serving, see our earlier notes on SageMaker KV-cache aware routing for vLLM endpoints and on MIG multi-instance GPU endpoints, which tackle utilization inside a Region rather than across Regions.

The vocabulary

table · 2 cols
TermMeaning
Inference routerA Regional SageMaker resource in your account that your app invokes
Routing targetA name for one logical model on a router, such as fraud-model; each fails over independently
DestinationA real-time endpoint, inference component or asynchronous endpoint, identified by ARN, which encodes its Region and account
Local destinationA destination in the router's own Region, preferred while healthy
Routing reasonA value returned with each real-time response explaining where the request went

A routing target holds either real-time destinations, which can mix endpoints and inference components across Regions and accounts, or asynchronous endpoints only. The Region where you create the router is its local Region; to prefer a different Region, create a router there.

How the router decides where a request goes

The logic, as AWS describes it:

  1. The router considers only destinations registered on the routing target that it is authorized to invoke.
  2. It prefers destinations in its own Region and spreads requests evenly across several healthy local ones.
  3. Requests leave the Region only when local destinations cannot serve reliably.
  4. SageMaker computes a health score per destination from recent requests, measured in the destination's own Region. Server errors (5xx), throttles and timeouts lower the score. Errors from the SageMaker platform, such as a failed health check, lower it the most. Errors caused by the request itself, such as invalid input, do not.
  5. Model latency is not an input, so an endpoint that keeps returning successful responses keeps its traffic.

Health is published to CloudWatch as one of three states: healthy (serves normally), overloaded (still serves, part of its traffic moves away) and degraded (failing, most traffic moves away, with a small share kept to detect recovery).

You do not tune thresholds. AWS manages them. If health information for remote destinations goes stale, the router keeps traffic in its own Region, so a monitoring gap never moves traffic on its own.

When traffic moves and when it returns

The more requests fail, the more traffic moves, so a destination need not fail outright first. Per the post, traffic starts to move within about a minute of sustained errors, or longer for a destination with little traffic. Traffic leaving the local Region is split across healthy remote destinations in proportion to their health, and you cannot set weights or a fallback order. After recovery, traffic returns over several minutes, more slowly than it left. Requests in progress are not interrupted.

Retries

When a destination fails a real-time request without running the model, for example a throttle, a 503, a model that is not ready, a rejection by a cross-Region traffic cap or access denied, the router retries once on a different destination. It never retries timeouts, 500 errors or model errors, because the model may already have run. For asynchronous requests it tries each eligible destination once, local first, until one accepts.

Setup in outline

AWS's example serves Qwen3-4B with the vLLM container as an inference component in US East (Ohio) and US West (Oregon). You create a router in the first Region with one routing target that lists both destinations by ARN, then confirm each destination's RoutingStatus is Routable with DescribeInferenceRouter. The application calls the router with the routing target name and the same payload it would send to InvokeEndpoint, and reads InvokedRoutedResourceArn and RoutingReason from the response.

In AWS's test, traffic stayed in us-east-2 with reason local-preferred while the local component ran. After the team removed that component, requests moved to us-west-2 with reason only-healthy, with no change to the application. After restoring it, traffic came back to us-east-2. A complete notebook is on GitHub, per the post.

The four routing reasons are local-preferred, capacity-available, failover and only-healthy. AWS recommends logging both the destination ARN and the reason with your own request logs.

Cross-account routing and permissions

A central platform account can own the router while other accounts own the endpoints. When the destination is in a different account:

  • The endpoint owner attaches a resource-based policy that allows the router account to invoke the endpoint. The sagemaker:SourceInferenceRouterArn condition key limits that access to one named router.
  • A second statement lets the router account call DescribeEndpoint; without it the destination shows an unknown status.
  • Callers in the router account need sagemaker:InvokeInferenceRouter on the router and sagemaker:InvokeEndpoint on every destination, because the router invokes each destination on the caller's behalf.

AWS stresses that a router never gives a caller access to an endpoint they could not call directly. For the identity-scoping principle behind that, compare our post on Bedrock Knowledge Base ACLs on AgentCore Gateway.

The cross-Region traffic cap

Failover has a cost for the receiving Region: overflow lands on endpoints that already serve local callers. By default a receiving endpoint accepts any amount of cross-Region traffic, so a surge can push it into throttling.

The owner can set CrossRegionTrafficConfig in CreateEndpointConfig with two values:

  • MaxInvocationsPerSecondPerInstance: what one instance can take, measured by a load test of your model, instance type and payload.
  • MaxCrossRegionPercentPerEndpoint: the share of capacity other Regions may use.

SageMaker multiplies the rate by instances in service and applies the percentage, so the cap follows autoscaling. AWS's example: 50 invocations per second per instance, a 20 percent cap and 4 instances gives 40 requests per second from other Regions; with 6 instances, 60. Only requests from other Regions count. Local calls are never throttled by the cap. Over-cap requests are rejected before the model runs, and the router retries each once elsewhere. If no destination can take it, the app gets a throttling error. The cap applies to real-time endpoints, and all inference components on an endpoint share it.

Monitoring

Per request, the response carries the serving destination and the routing reason. Asynchronous responses do not include the destination; use CloudWatch metrics to see placement. The router publishes OpenTelemetry metrics to CloudWatch, including invocations per destination, destination duration, router overhead duration, and destination status (healthy, overloaded or degraded). AWS suggests alarming on destinations that are not healthy and on overall error rate. CloudTrail records router API calls as management events and invocations as data events you can enable. Our coverage of SageMaker HyperPod enhanced observability shows the same direction on the training side.

Limits and planning notes

  • No streaming. If your app streams tokens, the router is not a fit yet.
  • About a minute. A real-time request not answered in about a minute fails with a service-unavailable error. Do not retry that automatically: the model may have run.
  • Quotas. Up to 20 routing targets per router, 10 destinations per target, 6 MB real-time payloads.
  • Capacity is yours. The router does not scale endpoints or copy models. A remote inference component with zero copies gets a small share of failover traffic and those requests fail until copies exist. Keep at least one running, or confirm autoscaling scales out.
  • The router is regional. If its Region is impaired, calls fail. AWS suggests routers in two Regions and client-side switching.
  • Data residency. A request may be processed in any Region where you registered a destination. Use separate routers per geography.
  • Clean up. The router never removes a deleted destination on its own.
  • Private access. Apps in a VPC can use an interface VPC endpoint (PrivateLink), IPv4 only.

What it costs

AWS states there is no additional charge for routing. You pay standard SageMaker AI pricing for the endpoints and inference components that serve requests, whether directly or through the router, and standard CloudWatch rates for router metrics. See SageMaker pricing for rates. The real cost is standby capacity: a second Region only helps if it has instances ready.

Who should adopt it

table · 2 cols
If youThen
Serve a latency-sensitive model on scarce GPU types in one RegionConsider a second Region behind a router
Stream tokens to usersWait; streaming is unsupported
Run a platform team serving many models across accountsOne router with a routing target per model fits
Have strict data residencyUse one router per geography

Bottom line

The router turns a pile of custom failover code into a managed resource, with health-based shifting, a single retry rule and a cap that protects the receiving side. It does not make GPUs appear. Treat it as insurance for capacity shortfalls, and budget for the standby capacity that makes the insurance work.

Related reading

  • SageMaker KV-cache aware routing for vLLM endpoints
  • SageMaker inference with NVIDIA MIG multi-instance GPUs
  • SageMaker HyperPod enhanced observability
  • Bedrock Knowledge Base ACLs on AgentCore Gateway
  • SageMaker serverless model customization with FFT
  • Bedrock AgentCore Identity with private CA

Details reflect the AWS announcement of November 19, 2026. Availability and quotas may change.

Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Nov 19, 2026

SageMaker HyperPod Enhanced Observability: Find the Bad GPU in a Training Job

One unhealthy GPU can stall a distributed training job, and the hard part is finding which one. AWS announced enhanced observability for SageMaker HyperPod on EKS and Slurm: outlier detection, static failure rules and CloudWatch as the new default metric store.

Nov 17, 2026

Amazon $17B Ohio AWS Data Centers: What Voter Bans Put at Risk

Ohio already passed the country densest data-center ballot season. A November 17 Financial Times analysis adds a dollar figure: more than $17 billion in Amazon Web Services spend across Ohio is now at risk where three proposed campuses hit new local limits.

Nov 17, 2026

Blackstone BXDC $5B Amazon Data Centers: Cap Rate vs 50% Cheaper Headline

Blackstone’s newly listed BXDC signed a $5 billion deal for majority stakes in 13 Amazon Web Services data-center shells in Indiana and Mississippi. Official documents describe a 7.0% initial cap rate and a 12-year AWS lease-back—not a 50% discount versus new construction.