NVIDIA CUDA 13.4 Memory-Enforced GPU Sharing: Fractioning Operator and KAI Scheduler Guide

Giving every small AI workload a whole GPU wastes expensive capacity, and until now sharing a GPU on Kubernetes meant picking between isolation and flexibility. NVIDIA's October 27, 2026 technical blog post describes a change in CUDA 13.4: a per-workload memory boundary that the application does not have to know about, plus an open-source operator and KAI Scheduler v0.18.0 that turn it into Pod annotations.

In plain terms: you can now tell a cluster "this container gets 40Gi of GPU memory, guaranteed, and may grow to 60Gi when neighbors are idle," and CUDA holds it to that. This guide covers what changed, the annotations, the two compute modes and the trade-offs, based on NVIDIA's post and the open-source gpu-fractioning repository.

TL;DR: questions teams ask first

Why GPU sharing was awkward before

NVIDIA lists three existing approaches. Multi-instance GPU (MIG) splits a GPU into hardware-isolated instances that are transparent to the application. Time-slicing lets several processes take turns but has no memory enforcement. Multi-Process Service (MPS) lets processes run concurrently in separate address spaces, the sm-sharing mode.

The gap is the middle: teams that cannot or do not want fixed MIG slices, but need a noisy neighbor not to take everyone's memory. CUDA 13.4 addresses that by enforcing the memory boundary itself, so "each workload sees only a fraction of the GPU memory," in NVIDIA's words.

What CUDA 13.4 gives a workload

NVIDIA lists four capabilities:

• An exact memory amount. A workload asks for a size and CUDA holds it there; going over fails.
• A range. A guaranteed minimum plus a maximum it can grow into when memory is free.
• A compute mode. Time-sharing by default, or sm-sharing per workload.
• A transparent limit. The application sees only its fraction, so no code changes.

This matters for inference servers and notebooks, which often need a memory floor to avoid out-of-memory kills but rarely need the whole device. It also connects to cost: idle GPU memory is the most common waste in shared clusters. Our coverage of Ai2's GPU cluster scheduler with time budgets and fair share shows the same pressure from the research side.

The missing piece: getting it from Kubernetes

CUDA provides enforcement, but Kubernetes cannot use it alone. On every GPU node something must run the daemons the features depend on, apply each container's limits before the process starts, and report per-fraction usage. That is the job of the GPU Fractioning Operator, which sets up and runs MPS on each node, applies memory limits and compute mode, and measures per-fraction use. It is independent of whichever scheduler placed the workload.

The contract between scheduler and operator is four per-container Pod annotations:

Here NAME stands for the container name. Any Kubernetes scheduler that can write these can enforce fractions. The scheduler still decides which GPU has room and never mixes compute modes on one device.

Using it with KAI Scheduler v0.18.0

KAI Scheduler is the first integration. From v0.18.0 it has a sharing mode called NvFractions, enabled at install with a Helm flag that NVIDIA shows as global.nvFractions.set=true. That also installs gpu-fractioning as a dependency. KAI's other sharing modes stay available; administrators choose one per cluster.

NVIDIA's examples describe two patterns:

1. Inference server across four GPUs. Request 40Gi per device, allow up to 60Gi, use sm-sharing, set gpu-fraction-num-devices to 4. The request applies to each device, so you get 40Gi on each of four GPUs.
2. Notebook or small fine-tune. A gpu-fraction of 0.5 with a limit of 0.8 on two devices, using the default time-slicing, so a crash does not affect neighbors.

Memory can be requested as an absolute size or as a portion of a GPU. The limit is optional; if omitted it equals the request, so the workload is capped at exactly what it asked for. A portion limit must be greater than the fraction and less than 1.

Time-slicing versus sm-sharing

With n workloads on a GPU, either mode gives each roughly 1/n of compute. The difference is how.

Because a GPU runs one mode at a time, KAI never places a time-slicing and an sm-sharing workload on the same GPU. It tracks this per GPU, so a node can host both modes on different devices.

What this means for what you build or pay

If you run shared GPU clusters, the practical win is higher utilization without rewriting applications. Inference services that only need 20-30Gi no longer need a full 80Gi device, and bursting lets them use spare memory when it exists. That can reduce the number of GPUs you rent or buy for the same workload mix, though NVIDIA's post gives no utilization or cost benchmark, so measure on your own traffic.

Trade-offs to weigh before adopting:

• Shared error domain in sm-sharing. One workload's fault can affect neighbors. Use it only among workloads you trust.
• Scheduler dependence. The feature works with KAI Scheduler today; other schedulers need to write the annotations.
• No published performance numbers. The post is an announcement and tutorial, not a benchmark.
• Burst is not a guarantee. Only the request is guaranteed; growth to the limit depends on free memory.

For agent and inference platforms, this pairs naturally with node health automation, covered in our post on NVIDIA Mission Control 2.5 autonomous recovery on Kubernetes, and with the capacity story in Vera Rubin NVL72's claimed efficiency numbers. At the other end of the scale, Apple's Minions paper tries to pool idle consumer GPUs, and a single desk-side box is covered in our DGX Spark MoE throughput write-up.

A grid of GPU chip squares with one row lit, showing fractional GPU allocation

What platform teams will ask next

Does this replace MIG? NVIDIA does not say so. MIG still gives hardware-isolated instances that are transparent to applications. The new mode targets teams that want software-enforced memory limits with finer, adjustable sizes and the option to burst. Which fits depends on how much isolation you need and how variable your workloads are.

What about observability? The operator is described as measuring memory and compute use per GPU fraction rather than per whole GPU. That matters for chargeback and for spotting workloads that request far more than they use, which is where shared clusters usually leak money.

Will existing code break? NVIDIA says the limit is transparent to applications, with an over-limit allocation failing as on a smaller GPU. In practice, services that assume they can grab all memory at startup, or that size caches from total device memory, are the ones to test first, because they now see only their fraction.

Is it only for NVIDIA's own scheduler stack? KAI Scheduler is the first integration, but the annotation contract is public, so other schedulers can write the same four keys. That is a design choice worth watching: the more schedulers adopt it, the less lock-in the feature carries.

How to try it

1. Read NVIDIA's admin and user guides in the repository for all three GPU sharing mechanisms.
2. Install KAI Scheduler v0.18.0 or later with NvFractions enabled in a test cluster.
3. Start with a time-slicing workload and a request-only memory annotation, then add a limit.
4. Run nvidia-smi inside the container to confirm it sees only its memory share.
5. Only then test sm-sharing with workloads you control.

NVIDIA also points to the community Slack and bi-weekly meetings, and to KubeCon plus CloudNativeCon NA 2026 in Salt Lake City, November 9-12.

Bottom line

CUDA 13.4 moves GPU memory limits from convention to enforcement, and the open-source operator makes them usable from Pod YAML. It will matter most to platform teams squeezing many small inference and notebook workloads onto shared GPUs. Treat the missing benchmarks as the open question: test utilization and tail latency yourself before betting a cluster on it.

Related reading

• Ai2 GPU cluster scheduler: time budgets and fair share
• NVIDIA Mission Control 2.5: autonomous recovery on Kubernetes
• Vera Rubin NVL72 claims: claimed vs verified
• Apple Minions: a virtual GPU cloud from idle Macs
• DGX Spark MoE throughput

Details are accurate as of October 27, 2026, based on NVIDIA's technical blog; check the repository for current versions.