Oriel — AI & technology worth noticing, every day.

Keep Distributed Training Running with Automated Workload Recovery

Web page · https://developer.nvidia.com/blog/keep-distributed-training-running-with-automated-workload-recovery

As distributed AI training scales, failures across hardware and software dependencies increasingly disrupt training progress, making fast, reliable recovery...

First seen by Oriel on .

AI summary

The post introduces NVIDIA Recovery, an open-source DSX OS service that detects, attributes, isolates, and restarts failed distributed training workloads to sustain cluster-wide goodput.

  • NVIDIA Recovery combines application logs, GPU metrics, and scheduler signals to detect crashes, hangs, NCCL and CUDA errors, interconnect failures, and stragglers.
  • Its control loop attributes failures, excludes implicated nodes, and restarts workloads from saved checkpoints, while configurable policies can stop unproductive retries.
  • Monitoring one cluster from July 9 to Oct. 9 covered nearly 200 jobs daily, detecting approximately 8,550 anomalies with mean detection and recovery-action latencies of 69 and 94 seconds.
  • The service integrates with Slurm through HTTP or gRPC and provides Helm-based in-cluster or SSH deployment; Kubernetes support is on the roadmap.
  • NVIDIA Recovery is Apache 2.0 licensed and supports monitor mode, a persistent audit trail, a Recovery Console, and workload opt-in through an sbatch directive.

#distributed training #slurm #ai operations #nvidia recovery

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
NVIDIA Developer Bloghttps://developer.nvidia.com/blog/?p=124491https://developer.nvidia.com/blog/keep-distributed-training-running-with-automated-workload-recovery/ Read discussion