SageMaker Inference Adds NVIDIA MIG Multi-Instance GPU: What It Changes for Endpoints

Amazon SageMaker AI can now partition GPUs on real-time inference endpoints that use inference components, using NVIDIA Multi-Instance GPU (MIG) to turn one accelerator into several right-sized units. AWS engineers published the details on November 6, 2026, including the AcceleratorPartitionLayouts and AcceleratorPartitionType fields, per-GPU profile tables for P4d, P5, G7 and G7e, Regional availability, and measured throughput and cost for Llama-3.1-8B-Instruct when a mig-3g.40gb slice of an H100 replaces a full A100.

TL;DR: what you need to decide

The problem AWS is solving

Until this change, the smallest GPU unit on a SageMaker endpoint was one whole accelerator. AWS gives three knock-on effects in its post:

1. Model count ceiling — an eight-GPU ml.p4d.24xlarge could host at most eight models if each needed its own accelerator, even when each model left most of the GPU idle.
2. Sizing mismatch — as concurrency and prompt length grow, a full GPU on an older instance can fall short, while a full GPU on a newer instance can be oversized for the same job.
3. DIY MIG on Kubernetes — teams that wanted partitions already ran their own stacks with drivers, device plugins and schedulers.

MIG splits one GPU into up to seven partitions on supported chips (A100, H100), each with fixed compute and memory. SageMaker now performs the carve at endpoint bring-up, maps inference components to named profiles, and keeps InvokeEndpoint addressing unchanged — you still call by component name.

How the API wires partitioning in

AWS stresses no new operations: partitioning reuses CreateEndpointConfig, CreateInferenceComponent, and friends.

1. AcceleratorPartitionLayouts on a production variant holds one or more layout entries. Each entry lists the MIG profile strings that will repeat on every accelerator in the variant, uniformly.
2. AcceleratorPartitionType inside an inference component’s ComputeResourceRequirements picks which profile that model occupies. With this field set, NumberOfAcceleratorDevicesRequired: 1 means one partition, not one physical GPU.

The blog’s JSON examples show a three-part layout on ml.p4d.24xlarge — mig-4g.20gb, mig-2g.10gb, mig-1g.5gb — which uses all seven A100 compute slices per GPU, repeated across eight GPUs for 24 slots of each profile type when CopyCount scales.

Operational implications AWS states plainly:

• Layout is immutable on a live endpoint — a new layout means a new endpoint and traffic move.
• Carving adds ~17 seconds to provisioning time in AWS testing before usual instance launch and container pull.
• Changing partitions is not an UpdateEndpoint tweak.

For the full field semantics and rejection cases, use Amazon’s GPU partitions with inference components guide alongside the blog, and cross-check AcceleratorPartitionLayouts on ProductionVariant and AcceleratorPartitionType on InferenceComponentComputeResourceRequirements in the SageMaker API reference.

Profiles by instance family

This table condenses the blog’s mapping — always confirm against NVIDIA’s published MIG tables for your driver line:

AWS also calls out a P4d driver footnote: mig-1g.10gb needs a newer driver than the default image may ship; the blog points to InferenceAmiVersion al2023-ami-sagemaker-inference-gpu-4-1 for the 580 driver line.

What AWS measured on Llama 3.1 8B

AWS ran genai-bench against two production SageMaker real-time endpoints with one model replica each:

• Baseline: one full A100-40GB on ml.p4d.24xlarge
• Candidate: one mig-3g.40gb partition on ml.p5.48xlarge

Both served Llama-3.1-8B-Instruct through the OpenAI-compatible SageMaker inference route with matched container, tokenizer and client configuration. Five traffic shapes swept concurrency from 2 to 512.

AWS chart of output tokens per second versus concurrency for H100 MIG partition versus full A100 on SageMaker inference Image: AWS

AWS chart of cost per million output tokens for MIG partitioned H100 versus full A100 across SageMaker traffic shapes Image: AWS

Headline outcomes from AWS’s narrative (not independent testing by explainx.ai):

• Throughput: the H100 partition delivered more output tokens per second in every scenario at every concurrency tested, 53–89 percent higher at the concurrency where cost benefit peaked, with lower inter-token latency (8.9–42.1 percent).
• Time to first token: lower on the partition in every case (5.6–47.5 percent).
• Cost per million output tokens: 18–34 percent lower above a crossover that moves earlier as prompts lengthen (example crossovers cited: concurrency 64 for 100-token prompts, concurrency 4 for 8,000-token prompts — below those points, full A100 was cheaper in AWS’s on-demand math).
• Stability: on 6,000- and 8,000-token loads, A100 missed deadlines for one third to one half of requests at the top concurrency step, while the partition recorded none — those panels end early in AWS figures because the baseline stopped reliably serving.

AWS priced units by dividing the on-demand hourly for ml.p4d.24xlarge ($25.251286/h) by eight GPUs and ml.p5.48xlarge ($63.296000/h) by sixteen GPUs, then attributing half of one H100 to mig-3g.40gb, explicitly marking the arithmetic as for analysis, not a promise of your invoice line item.

Salesforce’s architecture quote in the same post frames why enterprises care: many inference jobs do not need a whole GPU, and MIG is a way to consolidate safely without sacrificing predictability — a line of reasoning that matches how AWS positions inference components elsewhere.

When partitioning helps most

AWS lists six patterns:

1. Portfolios of small and mid models — classifiers, rerankers, embeddings.
2. Multi-tenant serving — hardware GPU boundaries, not full tenancy.
3. Multi-stage pipelines on one box — embed → rerank → generate without a network hop between stages.
4. Tiny always-on guardrails beside a big generator.
5. Challenger / canary models beside production without a second endpoint fleet.
6. Right-sizing onto newer silicon without paying for an entire H100 per small job.

Those line up with how teams already pack several models on one real-time endpoint. Our walkthrough of SeqKD distillation on SageMaker shows two inference components on the same host sharing base weights. MIG adds hardware-isolated slices inside each GPU when those components still do not fill the device.

Abstract counter window with a green light and ticket stream, illustrating queued requests at a SageMaker GPU inference endpoint

What builders ask before they carve GPUs

Do I lose throughput if I pick the wrong profile? Yes. A profile that fits the weight footprint but starves KV cache for your configured context length and concurrency will look fine in a smoke test and fail under production traffic — AWS calls that mistake out explicitly. AWS suggests adding weights, KV cache for your target context and concurrency, and 1–2 GB of runtime overhead before choosing a profile; for an 8B model at FP16 it cites about 16 GB of weights and a comfortable 8,000-token context on a 40 GB partition.

Is a partition “half an H100” in billing? No. AWS repeats that you still pay for the entire instance hour. The win is useful work per dollar of GPU silicon, not a smaller line item.

Can a model that needs more than one GPU use MIG? No. AWS says a model that needs a full accelerator, or more than one, should keep whole devices, and a partition never spans accelerators.

How does this relate to training? It does not replace training-session GPUs or Snowflake Cortex Training style loops; it changes how you host artifacts afterward. Keep the concerns separate in design docs so a training team does not “borrow” inference partitions for ad-hoc fine-tunes.

Limits and Region rollout

The blog is explicit about unsupported paths today:

• Serverless inference
• Heterogeneous instance pools
• Async invocation paths that cannot address inference components

Regional availability at launch: US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), Europe (Frankfurt), Europe (Ireland) — still confirm quota and instance stock in your account.

What teams should do this week

1. Model your KV cache alongside weights — AWS warns partitions sized only for FP16 weights fail once real concurrency arrives.
2. Pilot one layout on a non-production endpoint; exercise Create, invoke, and failure cases when a component asks for a profile not in the variant.
3. Compare on your tokenizer and traffic — AWS’s crossover points move with prompt and concurrency; short, sparse traffic may still favor whole A100s on their spreadsheet.
4. Plan migrations — because layouts cannot morph in place, bake blue/green cuts into the design.

For training-side GPU economics, pair this with our guide to SageMaker training sessions and custom RL loops and the multi-turn Wordle RL recipe: training and serving decisions still connect at the model artifact, even when the serving cluster is where MIG shows up.

Related reading

• SageMaker training session APIs and custom RL loops
• SageMaker multi-turn RL on GPT-OSS 20B
• SeqKD distillation shopping agent on SageMaker
• NVIDIA Reshard collective for RL weight refit
• SageMaker SDK v3 edge deployment patterns

Figures and numeric claims in the benchmark section follow Amazon’s November 6, 2026 SageMaker AI post. Validate pricing, quotas and profile names in your Region before production cuts.