explainx.ai350K+ learn from us
TrendingAI News TodayExplainx for Business
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • TL;DR: the 13 launches at a glance
  • Stage 1: choosing a configuration (launches 14 to 18)
  • Stage 2: fitting more models on each GPU (launches 19 and 20)
  • Stage 3: serving models larger than one instance (launches 21 to 23)
  • Stage 4: routing across Regions (launch 24)
  • Stage 5: keeping endpoints tuned (launches 25 and 26)
  • Which launches matter for which reader?
  • What is verified and what is AWS's claim
  • What teams are asking
  • Related reading
← Back to blog

explainx / blog

Amazon SageMaker Inference 2026 Launches, Part 2: 13 Features From MIG to In-Place Updates

Amazon SageMaker, AWS, LLM Inference, MLOps, GPU

Part of Microsoft, Apple and Amazon AI

AWS lists 13 more SageMaker inference launches (14-26): Studio recommendations, MIG, Dynamo, KV-cache routing, cross-Region routing and in-place updates.

Nov 20, 2026·8 min read·Yash Thakker
add explainx.ai
go deep
Amazon SageMaker Inference 2026 Launches, Part 2: 13 Features From MIG to In-Place Updates

AWS has published part 2 of its 2026 SageMaker inference review, adding 13 launches (numbers 14 to 26) and bringing the year's total to 26. The November 19, 2026 post, Amazon SageMaker Inference: 2026 launches in review, part 2, follows one fictional support team from a single 70B-parameter endpoint to multi-Region serving, and groups the launches by the problem they solve.

This is a vendor review, so the numbers are AWS's own. We separate what each launch does from the performance figures AWS attaches to it. Several of these features already have their own explainx.ai posts, linked below.

TL;DR: the 13 launches at a glance

table · 4 cols
#LaunchDateWhat it does
14Inference recommendations in SageMaker AI StudioJuly 2026Guided, low-code benchmarking and ranked configurations
15Recommendations in Python SDK v3August 2026Benchmark, recommend and deploy from notebooks or pipelines
16LoRA adapter support in recommendationsSeptember 2026Benchmarks adapter count against concurrency
17SLA-based concurrency searchSeptember 2026Finds the highest concurrency that meets latency and error targets
18Agent skill for inferenceOctober 2026Instance selection and benchmarking from coding agents
19Inference Components in the consoleOctober 2026Create, scale and monitor each model without scripts
20GPU partitioning with NVIDIA MIGOctober 2026Up to seven isolated partitions per GPU
21Managed NVIDIA Dynamo on HyperPodNovember 2026Multi-node and disaggregated serving, operated by HyperPod
22KV-cache-aware routingNovember 2026Routes to the instance holding the longest cached prefix
23HyperPod Inference GatewaySeptember 2026One OpenAI-compatible endpoint per cluster, GPU-aware routing
24Cross-Region Inference Routernot datedRoutes across Regional and cross-account endpoints
25Continuous performance monitoringNovember 2026Detects latency drift, configuration gaps, idle GPUs
26In-place rolling model updatesNovember 2026Updates a fleet with one batch of extra instances
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

Stage 1: choosing a configuration (launches 14 to 18)

AWS's example team picked its first configuration from an April benchmark. Since then it added four LoRA adapters and set a 500-millisecond target for time to first token (TTFT) at peak load. The old benchmark never tested either change.

The Studio workflow lets you choose a use-case profile (Interact, Generate, Summarize or Custom), a goal (minimise latency, maximise throughput or minimise cost) and a model from JumpStart, Amazon S3, the Model Registry or an existing model. AWS says SageMaker benchmarks candidates on real GPU infrastructure with NVIDIA AIPerf and applies techniques such as speculative decoding for throughput or kernel tuning for latency. Results come back ranked by TTFT, inter-token latency, throughput and cost.

Two additions are the most practical. LoRA support accepts adapters as Model Package ARNs or S3 URIs, including adapters from Hugging Face PEFT, and recommends deploying them as separate inference components so you can add or remove one without touching the base model. The SLA-based search takes a search_recipe of max-concurrency-under-sla plus targets for TTFT, time per output token, end-to-end latency and error rate, and returns the largest passing concurrency (feasible_max) or the smallest failing one (infeasible_min). That replaces manual sweeps.

The agent skill, aws-ai-ml in the Agent Toolkit for AWS, gives coding agents such as Kiro SageMaker inference knowledge. AWS says setup takes about 10 minutes and it works with any agent that supports the Model Context Protocol. For background on that protocol, see our MCP guide.

Stage 2: fitting more models on each GPU (launches 19 and 20)

SageMaker inference launches: three small models sharing partitions of one GPU with MIGSageMaker inference launches: three small models sharing partitions of one GPU with MIG Image: AWS

The team also runs an embedding model, a reranker and a small classifier that each use a fraction of a GPU. Inference Components, which host several models on one endpoint, now appear in the SageMaker AI console. You can create and edit components, set an automatic scaling policy per component down to zero copies, and see per-component alarms and metrics.

MIG divides one physical GPU into as many as seven partitions with dedicated compute and memory and hardware isolation. Applications still invoke models by inference component name. The caveat matters for budgeting: models on one instance still share host CPU, host memory, networking, storage and endpoint routing, and you pay the standard rate for the full instance-hour. The saving is in placement, not in price. We covered this in SageMaker Inference Adds NVIDIA MIG.

Stage 3: serving models larger than one instance (launches 21 to 23)

When a reasoning model's weights exceed any single instance's GPU memory, AWS moves the workload to HyperPod Inference. Three launches apply.

Managed NVIDIA Dynamo gives multi-node and disaggregated serving with HyperPod operating the platform underneath. For the engine itself, see NVIDIA Dynamo and KV-aware routing for RL rollouts.

KV-cache-aware routing has the container publish cache creation and eviction events to a central index, and SageMaker routes each request to an eligible instance holding the longest matching cached prefix. It works with single-model endpoints and inference components, including LoRA adapters, and supports InvokeEndpoint, InvokeEndpointWithResponseStream and the OpenAI-compatible Completions and Chat Completions payloads. The first release requires an AWS-verified container. Prefix-aware routing, from part 1, needs no container changes. Read our deeper post on KV-cache-aware routing for vLLM endpoints and the explainer what is a KV cache.

The HyperPod Inference Gateway installs with the Inference Operator add-on and gives each cluster one OpenAI-compatible endpoint for all models. It weighs KV cache utilisation, queue depth, running requests, prefix cache hit rate and LoRA adapter residency. AWS says it cut P99 TTFT by up to 97% on mixed GPU generations and up to 98% under bursty traffic versus round-robin. Those are AWS's own comparisons against the weakest baseline, so expect smaller gains against a tuned router.

Stage 4: routing across Regions (launch 24)

The Cross-Region Inference Router groups endpoints that serve the same model under one routing target, in the same or different accounts. It tries the local endpoint first and moves traffic when it cannot serve a request. It does not add capacity, replicate models or deploy endpoints. Traffic uses AWS PrivateLink with cross-account authorisation, the endpoint owner can cap overflow traffic, and the application can restrict which Regions may serve a request. See SageMaker Cross-Region Inference Routing for the failover details. AWS's own post on the feature is here.

Stage 5: keeping endpoints tuned (launches 25 and 26)

Two months after the team adds retrieval, average prompt length roughly doubles and TTFT rises, yet no alarm fires. Continuous performance monitoring catches this class of problem. AWS says it is on by default with no setup and opt-out per endpoint, looks only at latency, GPU utilisation and serving configuration, not your requests or responses, and reports three finding types: latency drift after a traffic change, a configuration gap present since deployment, and idle GPU capacity. Nothing deploys until you apply it.

In-place rolling update reusing the fleet instead of acquiring new GPU instances at every stepIn-place rolling update reusing the fleet instead of acquiring new GPU instances at every step Image: AWS

In-place rolling updates attack a different pain: a standard rolling update finds new capacity at every step, and can fail partway when the next batch of GPU instances is unavailable. AWS now provisions one small batch, shifts traffic, and reuses freed instances. With 20 instances and batch size 2, AWS says the update acquires 2 new instances instead of 20. AWS plans scale-in-first updates in early 2027 that would remove the need for any extra instances.

Which launches matter for which reader?

table · 2 cols
If you are...Look first at
Running one LLM endpoint with traffic that has changedRecommendations with SLA search (17) and monitoring (25)
Hosting many small modelsInference Components console (19) and MIG (20)
Serving a model too big for one instanceManaged Dynamo (21), KV-cache routing (22), Inference Gateway (23)
Short of GPU capacity in one RegionCross-Region Router (24) and in-place updates (26)
A coding-agent userThe aws-ai-ml agent skill (18)

What is verified and what is AWS's claim

The feature descriptions come from AWS's own post and linked developer documentation. The performance figures (97% and 98% lower P99 TTFT, 2 versus 20 instances) are AWS's examples. We have not run them. The review is also a marketing roundup: it ends by pointing to the AWS Experience-Based Acceleration program. Treat the list as a map of what exists, then test each feature on your own workload. Pricing details sit on the SageMaker pricing page.

For related AWS inference and agent work, see SageMaker HyperPod enhanced observability and SageMaker fine-tuning with serverless customization.

What teams are asking

Do I need to change client code for MIG? AWS says applications keep invoking models by inference component name, so client code does not change.

Does cross-Region routing replicate my model? No. You must already have endpoints deployed; the router only sends traffic between them.

Will in-place updates need a config change? AWS says customers do not need to change anything.

Related reading

  • SageMaker Inference Adds NVIDIA MIG
  • SageMaker KV-cache-aware routing for vLLM endpoints
  • SageMaker Cross-Region Inference Routing
  • SageMaker HyperPod enhanced observability
  • NVIDIA Dynamo and KV routing for RL rollouts
  • What is a KV cache?
  • What is MCP?

Launch names, dates and figures are as stated in AWS's November 19, 2026 post and may change.

Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Oct 27, 2026

SageMaker SOCI-Indexed Images: Training Containers Start in Seconds, Not Minutes

On October 27, 2026 AWS announced Seekable OCI (SOCI) indexing for SageMaker AI training and processing jobs. It lazy-loads container images so jobs start in seconds. We cover the numbers, the two ways to adopt it, and when it will not help.

Oct 26, 2026

One Model per Camera: AWS Shows a SageMaker SDK v3 Pipeline for Edge Vision Fleets

A fleet of factory cameras rarely shares one model. An AWS Machine Learning blog post from October 26, 2026 shows a SageMaker SDK v3 reference pipeline that trains an anomaly detector and a classifier per camera, gates them on metrics suited to imbalanced defect data, compiles for the edge with Neo, and paces thousands of retrains.

Nov 19, 2026

SageMaker Cross-Region Inference Routing: Failover Without Writing Your Own Router

AWS made cross-Region inference routing generally available in SageMaker AI on November 19, 2026. A new inference router sends requests to a healthy endpoint in another Region or account when the local one runs short of GPUs. Here is how it decides, what it costs and where it stops.