NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes GPU Clusters

NVIDIA has released Mission Control 2.5, and its main change is that the autonomous recovery engine now works for Kubernetes workloads managed by NVIDIA Run:ai. Until now that engine covered Slurm-based HPC jobs. According to the NVIDIA Technical Blog post published October 26, 2026 by Ryan Prout and Casey Dugas, the goal is to move AI factories from reactive maintenance to self-healing infrastructure: detect what failed, isolate and tag the bad hardware, and get the workload back onto healthy compute fast.

The post is a design walkthrough, not a benchmark report. It includes no recovery times or utilization gains. This guide explains how the loop works, walks through the three failure scenarios NVIDIA describes, and lists what an operator should still verify. It sits next to NVIDIA's other operations launches this month, including Minerva, the AIOps observability platform, and the open-sourced KAI scheduler behind Run:ai.

TL;DR

Why recovery automation matters at cluster scale

A large training run is a chain: every GPU must stay healthy and in sync, so one bad node can stall hundreds of others. Operators know the pattern. A GPU throws ECC errors, a worker hangs, or a collective communication call times out. A human sees the alert, finds the node, cordons it, restarts the job, and opens a ticket. Each step costs minutes or hours of accelerator time that nobody is using.

NVIDIA frames this with a simple matrix in its post. One axis is node health, from returning errors to operating as expected. The other is scheduler availability, from available to out of scheduling. The four quadrants are:

• Productive Capacity: healthy and schedulable, the target state.
• Workloads at Risk: needs maintenance but is still schedulable, so jobs can land on it.
• Safely Isolated: returning errors and temporarily removed from circulation.
• Stranded Capacity: healthy but unavailable to workloads, which wastes money.

NVIDIA's claim is that recovery automation should push nodes out of the risky and stranded states and back into productive capacity as quickly as possible. The placement of quadrants is inferred from the axis labels and caption, so check the original figure if you quote it. For the economics behind why idle GPUs hurt, see our look at the neocloud shakeout and rental pricing pressure.

How the three-layer loop works

The engine splits responsibility into three parts.

1. Kubernetes scheduler (placement). Run:ai is the default scheduler in Mission Control. It owns AI workloads, queues and policies, and places jobs onto available capacity.
2. Autonomous job recovery (workload context). It tracks running jobs and where they run, uses failure history and job topology to pick a recovery path, and works with the resource manager to restart, retry, requeue or change placement.
3. Autonomous hardware recovery (infrastructure health). It watches nodes and GPUs, removes suspect hardware from scheduling, and runs diagnostics or remediation.

The authors argue the separation is what makes the model portable: "Keeping these responsibilities separate is what makes the model portable." Job recovery does not need to fix hardware, hardware recovery does not need to understand each training framework, and the scheduler can use ready capacity without waiting for root-cause analysis. That last point is the practical gain. The job restarts first; the post-mortem happens in parallel.

The same split explains the Kubernetes expansion. Slurm and Run:ai are different schedulers with different APIs, but the job and hardware services sit above both. Mission Control 2.5 therefore offers platform teams a single recovery model across Slurm and Kubernetes environments, according to NVIDIA. If you want background on Run:ai itself, our piece on KAI scheduler going open source covers the scheduling side.

Scenario 1: a hardware fault during training

In the first scenario a GPU shows ECC errors, a worker hangs, or a distributed run hits a communication timeout. NVIDIA describes two parallel tracks:

• Workload recovery. The job recovery service captures the job context and identifies the suspect node or GPU. The scheduler integration then restarts or reschedules the job on healthy GPUs, and training resumes.
• Hardware recovery. The hardware service isolates the node, records why, and starts diagnostics. That work continues after the job is already running again.

The post also describes a seven-step breakfix workflow. In the protect-and-audit phase the faulty node is made unschedulable, labeled as a breakfix candidate, and annotated with a reason so there is an audit trail. A poller then notices nodes that left scheduler service and starts automated triage: it verifies agent connectivity, sets a maintenance lock so two systems do not work on the same node, and launches hardware diagnostics. The benefit NVIDIA names is lower downtime, because the workload never waits for root-cause analysis and the broken node is not handed to the next job.

Scenario 2: a transient software fault

Not every failure is hardware. A worker may exit, hit a recoverable timeout, or briefly lose coordination while the node is perfectly healthy. Here the job recovery service reads the job context so it does not overreact. If the failure history looks transient, it retries under scheduler policy, typically restarting from a checkpoint. If the failure repeats or is non-retryable, the flow takes an off-ramp and surfaces the issue to operators.

Meanwhile the hardware recovery service keeps monitoring but stays out of the recovery path unless new evidence points at hardware. NVIDIA calls the benefit restraint: the workload recovers without operator noise and without needlessly shrinking the scheduling pool. For teams that have lived through alert storms, this is arguably the more important design decision. A recovery system that cordons healthy nodes for every software hiccup quietly destroys capacity.

Scenario 3: catching degradation before a job lands

The third case is preventive. A node shows repeated GPU health warnings, connectivity problems or other signs it is risky to schedule on. The hardware recovery service removes it from scheduler eligibility before a training job lands there and starts diagnostics. Healthy capacity stays schedulable and new jobs avoid suspect nodes. If health checks fail, the node is marked unschedulable and sent to diagnostics and remediation.

In the best case the job recovery service never has to act, because the faulty capacity left the pool before it disrupted anything. This is the closest Mission Control 2.5 gets to the prediction-style features in Minerva, which builds a knowledge graph of GPUs, switches and jobs to find slow-burn problems. The two products are complementary on paper: Minerva is observability and incident grouping, while Mission Control acts on the scheduler and hardware.

What else shipped in 2.5

• AI-enabled network anomaly detection (beta). A new service for spotting network anomalies, available in beta only. NVIDIA gives no detail on models or accuracy in the post.
• Central Keycloak authentication. Authentication across services now runs through a central Keycloak instance.
• Security and Kubernetes hardening. Additional updates, unspecified in the post.
• Early access for NVIDIA Vera Rubin NVL72. The platform is available in early access. For where Vera Rubin capacity is landing, see Google Cloud's A5X Vera Rubin NVL72 rollout and NVIDIA's DSX pod announcement.

The full list of changes lives in the 2.5.0 release notes and the Mission Control documentation hub, which is where NVIDIA sends readers for upgrade guidance.

Claimed versus measured

NVIDIA's post argues for fewer stalled hours and better accelerator utilization, but it does not quantify either. Here is how the claims stack up.

None of this means the design is wrong. Separating placement, job context and hardware health is a sensible architecture, and the checkpoint-restart pattern is standard. It means buyers should ask for pilot data. Useful questions include: how are checkpoints discovered for non-standard frameworks, how does the service avoid restart loops when a failure is actually in user code, and what happens when a large fraction of a cluster is cordoned at once during a rack-level event.

What this means if you run or rent GPU capacity

Platform teams on Kubernetes. If you already schedule through Run:ai, 2.5 is the version to evaluate, ideally on a staging pool first. Test fault injection: pull a GPU, kill a worker, saturate a link, and time the loop. Check the audit annotations on cordoned nodes so your on-call rotation understands why a node vanished.

Slurm shops moving to Kubernetes. The promise of one recovery model across both is attractive during a migration, since you do not have to rebuild runbooks twice. Confirm which recovery behaviors differ.

Teams renting from neoclouds. You cannot install Mission Control on someone else's cluster, but you can ask whether the provider uses it, how quickly failed nodes are replaced, and whether the contract measures goodput or just GPU-hours. With rental prices under pressure, operational quality is a differentiator.

Builders training models. Recovery on the infrastructure side does not remove the need for frequent, tested checkpoints. The engine restarts from a checkpoint; if yours is hours old, you still lose hours.

Limits and open questions

• The post does not list system or software requirements, so check the release notes before planning an upgrade.
• There are no published recovery-time, uptime or utilization numbers.
• The network anomaly service is beta, and NVIDIA offers no accuracy data.
• All evidence is from NVIDIA's own write-up. No independent operator has published results yet in the sources we checked.
• Vera Rubin NVL72 support is early access, not general availability.

Bottom line

Mission Control 2.5 is a meaningful step for operators on Kubernetes: the same job-and-hardware recovery logic that served Slurm clusters now covers Run:ai workloads, with sensible restraint around software faults. It is also a design announcement, not a performance claim, and the first customer numbers will decide how much it matters. We will update this post if NVIDIA or a customer publishes measurements.

Details reflect NVIDIA's October 26, 2026 post and may change as the documentation is updated.

Related reading

• NVIDIA Minerva: AIOps for AI factories
• NVIDIA open-sources the KAI scheduler (Run:ai)
• NVIDIA DSX pod: containerized Vera Rubin AI factory
• NVIDIA OCP Vera Rubin MGX open designs and grid resilience
• Google Cloud A5X Vera Rubin NVL72 availability
• Neocloud shakeout and Nvidia exposure
• GTC Berlin 2026 keynote: everything announced
• Official: NVIDIA Mission Control product page