Oriel — AI & technology worth noticing, every day.

How NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes

Web page · https://developer.nvidia.com/blog/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes

Training large-scale AI models on thousands of GPUs is a high-stakes operation where monitoring and automation must be built-in to optimize cluster resiliency...

First seen by Oriel on .

AI summary

NVIDIA Mission Control 2.5 extends its autonomous recovery engine to Kubernetes workloads managed by NVIDIA Run:ai, coordinating job recovery and hardware remediation to keep AI training productive.

  • The recovery loop combines the Kubernetes scheduler, autonomous job recovery, and autonomous hardware recovery, with each service handling workload placement, job context, or infrastructure health.
  • For suspected hardware faults, jobs can restart or reschedule on healthy nodes while the problematic node is isolated and diagnostics continue.
  • Transient software failures can trigger scheduler-policy retries without unnecessarily quarantining healthy hardware; repeated failures are surfaced.
  • Hardware recovery can remove degraded nodes from scheduling before they disrupt workloads, then return them after validation.
  • Mission Control 2.5 is available today and also includes beta network anomaly detection, authentication improvements, security and Kubernetes hardening, and early access for NVIDIA Vera Rubin NVL72.

#kubernetes #nvidia run:ai #ai infrastructure #fault recovery

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
NVIDIA Developer Bloghttps://developer.nvidia.com/blog/?p=123117https://developer.nvidia.com/blog/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes/ Read discussion