The announced update brings NVIDIA’s recovery tools to Kubernetes workloads managed through Run:ai, but independent performance results are not yet provided.
NVIDIA says Mission Control 2.5 can recover Kubernetes-based AI workloads in two separate tracks: one for the job and another for the hardware.
The update extends Mission Control’s autonomous recovery engine beyond Slurm-based high-performance computing workloads. NVIDIA says the system now also supports Kubernetes workloads managed with NVIDIA Run:ai. Mission Control 2.5 is available today for existing Mission Control users.
One failure, two recovery paths
When a training job fails, the job recovery service examines its context, failure history and topology—the way its tasks are spread across machines. It can then guide a restart, retry, re-queue or move to different capacity through the resource manager.
If the failure points to a GPU or node, the hardware recovery service handles the infrastructure separately. NVIDIA says it monitors nodes and GPUs, removes suspicious resources from scheduling, and begins diagnostics or remediation.
That separation matters during a hardware fault. A job can restart or move to healthy nodes while the suspect hardware remains isolated and under investigation. The scheduler does not have to wait for a full root-cause diagnosis before moving the workload.
For a temporary software problem, NVIDIA says the system can retry the job without unnecessarily quarantining healthy hardware. It can also remove a degrading node before another job is placed there.
The practical promise is less manual triage and fewer repeated failures in shared GPU clusters. However, NVIDIA’s announcement does not provide independent recovery-time benchmarks, compute-cost results or customer deployments. Pricing, licensing terms and supported Kubernetes versions also remain important questions for teams considering adoption.