Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod
Silent data corruption is a GPU fault that returns wrong but plausible numbers with no error, quietly poisoning distributed training. This post adds three checks to a PyTorch DDP loop to detect it, localize the faulty node on Amazon SageMaker HyperPod, and hand the node to automatic recovery for replacement.
First seen by Oriel on .
AI summary
AWS presents three training-loop checks for detecting, localizing, and recovering from silent data corruption in PyTorch DDP workloads on Amazon SageMaker HyperPod.
- Non-finite gradient checks detect NaN, Inf, and overflow before and after all-reduce, while ring replay audits recompute another node’s micro-batch to catch finite errors.
- Weight fingerprints compare replicas after optimizer updates, detecting corruption in optimizer steps or stored weights, but not shared compute-path errors or sharded weights.
- AWS tested software-injected faults on four ml.p5.48xlarge instances with 32 NVIDIA H100 GPUs; the checks require at least three GPU nodes.
- Recovery maps the suspect rank to an EKS node, stops the job, cordons and investigates the node, replaces it, then resumes from a pre-corruption checkpoint.
- Fingerprints use plain and position-weighted 64-bit sums; equality is strong evidence rather than proof because sufficiently large changes could theoretically cancel out.
#amazon sagemaker #pytorch ddp #silent data corruption #distributed training
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| AWS Machine Learning Blog | 843b103fd6b2a85ee84c6248187646fe49551f31 | https://aws.amazon.com/blogs/machine-learning/detecting-and-recovering-from-silent-data-corruption-on-amazon-sagemaker-hyperpod/ Read discussion |