Keep Distributed Training Running with Automated Workload Recovery
Web page · https://developer.nvidia.com/blog/keep-distributed-training-running-with-automated-workload-recovery
As distributed AI training scales, failures across hardware and software dependencies increasingly disrupt training progress, making fast, reliable recovery...
First seen by Oriel on .
AI summary
The post introduces NVIDIA Recovery, an open-source DSX OS service that detects, attributes, isolates, and restarts failed distributed training workloads to sustain cluster-wide goodput.
- NVIDIA Recovery combines application logs, GPU metrics, and scheduler signals to detect crashes, hangs, NCCL and CUDA errors, interconnect failures, and stragglers.
- Its control loop attributes failures, excludes implicated nodes, and restarts workloads from saved checkpoints, while configurable policies can stop unproductive retries.
- Monitoring one cluster from July 9 to Oct. 9 covered nearly 200 jobs daily, detecting approximately 8,550 anomalies with mean detection and recovery-action latencies of 69 and 94 seconds.
- The service integrates with Slurm through HTTP or gRPC and provides Helm-based in-cluster or SSH deployment; Kubernetes support is on the roadmap.
- NVIDIA Recovery is Apache 2.0 licensed and supports monitor mode, a persistent audit trail, a Recovery Console, and workload opt-in through an sbatch directive.
#distributed training #slurm #ai operations #nvidia recovery
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| NVIDIA Developer Blog | https://developer.nvidia.com/blog/?p=124491 | https://developer.nvidia.com/blog/keep-distributed-training-running-with-automated-workload-recovery/ Read discussion |