NVIDIA has released NVIDIA Recovery, an open-source service that finds crashed, hung and slow distributed training jobs and restarts them without a human. The NVIDIA Technical Blog post is dated November 2, 2026. The code is in the dsx-ai-factory/NVIDIA-Recovery repository under Apache 2.0.
The repo has a caveat the blog post does not repeat. Its README says: "NVIDIA Recovery is experimental and not ready for production use." This post covers how the recovery loop works, what the numbers mean, and how to try it safely.
At a glance
| Question | Answer |
|---|---|
| What is it? | A cluster-level service that automates recovery of training jobs |
| Part of | NVIDIA DSX OS, the software layer of the DSX AI Factory Platform |
| License | Apache 2.0 |
| Schedulers | Slurm and Slurm on Kubernetes (Slinky). Kubernetes-native is not implemented yet |
| Hardware | NVIDIA GPU clusters from Ampere through Hopper to Blackwell (per the README) |
| Default mode | Monitor only. No scheduler actions until you enable them |
| Reported numbers | About 8,550 detections, 69 s mean detection, 94 s mean recovery action |
| Status | Experimental, per the repo README |
Why training jobs need a recovery service
Large training runs fail often, and each failure idles expensive GPUs. A stalled rank can cause an NCCL timeout. A node can stop answering the scheduler. One slow GPU, a straggler, can drag down the whole job.
NVIDIA gives a simple calculation. If each GPU survives a day with 99.9% probability, a job on 10,000 GPUs finishes that day uninterrupted with a probability of about 0.005%. The post also cites an 11-month study of 24,000 A100 GPUs at Meta. In that study, hardware failures interrupted fewer than 1% of jobs, but those jobs used 19% of GPU runtime. The biggest, longest jobs are the most exposed.
Manual recovery means someone reads logs, finds the bad node, and resubmits from a checkpoint. At 3 a.m. that can take hours. NVIDIA calls the result lost "goodput", the share of wall-clock time that becomes real training progress.
How the recovery loop works
NVIDIA describes four stages. The exact action depends on the failure and your policy.
- Detect. The Anomaly Detector reads application logs and scheduler signals. It recognizes PyTorch and NCCL failures, CUDA errors and interconnect failure patterns. Missing progress signals or known communication-hang patterns flag a hang. Sustained throughput below a recent baseline flags a straggler.
- Attribute. The service classifies the interruption. With the optional NVIDIA Attribution component, it separates hardware faults, system-software issues and application bugs. You can plug in your own attribution service instead.
- Isolate. The Anomaly Remediator excludes implicated nodes from the next restart. It tracks nodes across attempts so a bad node does not keep returning. A watchdog lets normal scheduling continue if a recovery decision takes too long or fails.
- Restart or stop. The scheduler integration acts on the recommendation. The job resumes from its last saved checkpoint using its own restart logic. If a job fails to save a new checkpoint across a configurable number of attempts, retries stop.
The README stresses one design point. The service records a decision as "Advice" before anyone acts on it. That keeps each decision inspectable. If the controller restarts mid-recovery, it checks the current scheduler state first, so it does not repeat finished actions.
Where it sits in NVIDIA's resilience stack
NVIDIA argues that workload recovery is a separate problem from other layers. The post names these pieces:
- NVSentinel monitors GPU health and XID events on Kubernetes and cordons faulty nodes.
- Slurm prolog and epilog scripts check node health at many Slurm sites. The Slinky slurm-operator injects the same scripts.
- NVRx gives in-job recovery, hang detection and straggler mitigation, but each user must configure it.
- NVIDIA Recovery coordinates across jobs: restarts, placement and node exclusion.
The layers have no hard dependencies. If you already run NVSentinel, you can add Recovery with little change. If you run NVRx, Recovery covers the cross-workload gap that an in-job restart cannot reach.
This builds on earlier NVIDIA work. Our coverage of NVIDIA Mission Control 2.5 explains the infrastructure side: finding faulty nodes and repairing them. Recovery connects that to the training job itself.
Grid of GPU chip squares with one row lit, showing cluster scheduling for NVIDIA Recovery workload restarts
The production numbers, read carefully
NVIDIA reports that from July 9 to October 9, Recovery monitoring on one training cluster covered nearly 200 jobs per day. It recorded about 8,550 anomaly detections. Mean detection latency was 69 seconds. Mean recovery-action latency was 94 seconds.
Three cautions apply.
- It was one cluster. The post does not name it or say how many GPUs it has.
- "Detections" are not "saved runs". The post does not say how many detections were real failures, false alarms or successful restarts.
- The latency stops early. NVIDIA states that recovery-action latency "ends at the automated action." Scheduling and checkpoint restoration decide when training actually resumes. The post gives no time-to-resume figure.
These are vendor-reported figures with no independent test. They show the service ran at scale in monitoring. They do not yet prove how much goodput it returns.
Slurm integration and monitor mode
Recovery runs as a control-plane service and talks to a scheduler through an HTTP or gRPC API. For Slurm, NVIDIA ships a reference Advisor Controller as a Helm chart. It has two modes:
- In-cluster mode runs the controller inside the cluster.
- SSH mode connects remotely to Slurm without adding an in-cluster service. This helps when a new component needs a change window or security review.
New deployments start in monitor mode. The service publishes advice and records what it would have done. You compare its decisions with what your engineers would have done, then enable action mode. NVIDIA says IBM Research took this path on its Slurm cluster and enabled autonomous action after validating the signal. Verda, a neocloud provider, was the first NVIDIA Cloud Partner to deploy it. It ran Recovery with VictoriaLogs and has submitted its first pull requests.
The audit trail records each detection pattern, the action (kill, requeue, release or node exclusion), the timestamp, the workload ID and the attribution verdict. A Recovery Console shows live workload state and a per-workload timeline.
How to try it
NVIDIA lists three steps:
- Install the Helm umbrella chart. A bundled MySQL sub-chart lets you run the quickstart locally with no database and no physical GPU. For production, point it at managed MySQL.
- Validate your signals. Point the chart at your Loki or Victoria Logs endpoint and your Prometheus target. Run the log ingestion configurator first. It tests your signals against the Recovery contract and finds label mismatches.
- Opt a workload in. Add one line to your sbatch script:
#SBATCH --comment='{"nvidia-recovery":{"environment":"prod","mode":"observe"}}'
To move from advice to action, set mode to action and acknowledgeSchedulerActions to true in the Helm values.
Your training script needs to be checkpoint-capable. Recovery restarts the job; the job must resume itself. The README also says log-based detection needs configured patterns and progress signals, and that straggler-triggered recovery is opt-in.
Limits to know before you deploy
- Experimental. The README says not to use it in production yet.
- No Kubernetes-native recovery. JobSet and Kubeflow PyTorchJob support is on the roadmap. Slurm job arrays come next.
- Attribution source is not public. The README says NVIDIA Attribution is enabled by default but its source is not yet publicly fetchable. A degraded OFF mode exists, and the local demo uses it.
- You build your own images. The project ships source and Helm charts only.
- Scope. It manages workloads. It does not repair hardware or store checkpoints.
- Early traction. The repository showed 18 stars and 12 commits when we read it, so expect rough edges.
What this means for teams that train models
If you run your own Slurm cluster, the useful idea is the monitor-first pattern. Run the detector for a few weeks. Compare its advice with your on-call decisions. Only then let it act. That approach limits the risk of a wrong automatic restart wasting a good run.
If you rent GPUs from a cloud, expect providers to build similar loops. Recovery speed is now part of what you buy, next to price per GPU-hour. Our look at Ai2's GPU cluster scheduler shows another team treating cluster time as a scarce budget. For smaller setups, the eight-node DGX Spark cluster guide shows the scale where manual recovery still works.
The pressure grows with hardware. NVIDIA's rack-scale systems pack more GPUs into each job, as we covered in Vera Rubin NVL72 claims versus verified facts and in the AICR Kubernetes recipes for Vera Rubin. More GPUs per job means more chances that one fails. Moving weights quickly after a restart also matters, which the weight teleportation write-up touches on for RL training.
Summary
NVIDIA Recovery packages a real operations problem into an Apache 2.0 service. Its design is careful: advice first, audit trail, watchdog, monitor mode by default. The reported 69-second detection and 94-second action times are promising but narrow. The README's "experimental" label is the most honest guide. Test it in monitor mode on a non-critical cluster, and wait for a time-to-resume figure before you trust it with a flagship run.