Oriel — AI & technology worth noticing, every day.

Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

Web page · https://aws.amazon.com/blogs/machine-learning/detecting-and-recovering-from-silent-data-corruption-on-amazon-sagemaker-hyperpod

Silent data corruption is a GPU fault that returns wrong but plausible numbers with no error, quietly poisoning distributed training. This post adds three checks to a PyTorch DDP loop to detect it, localize the faulty node on Amazon SageMaker HyperPod, and hand the node to automatic recovery for replacement.

First seen by Oriel on .

AI summary

AWS presents three training-loop checks for detecting, localizing, and recovering from silent data corruption in PyTorch DDP workloads on Amazon SageMaker HyperPod.

  • Non-finite gradient checks detect NaN, Inf, and overflow before and after all-reduce, while ring replay audits recompute another node’s micro-batch to catch finite errors.
  • Weight fingerprints compare replicas after optimizer updates, detecting corruption in optimizer steps or stored weights, but not shared compute-path errors or sharded weights.
  • AWS tested software-injected faults on four ml.p5.48xlarge instances with 32 NVIDIA H100 GPUs; the checks require at least three GPU nodes.
  • Recovery maps the suspect rank to an EKS node, stops the job, cordons and investigates the node, replaces it, then resumes from a pre-corruption checkpoint.
  • Fingerprints use plain and position-weighted 64-bit sums; equality is strong evidence rather than proof because sufficiently large changes could theoretically cancel out.

#amazon sagemaker #pytorch ddp #silent data corruption #distributed training

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
AWS Machine Learning Blog843b103fd6b2a85ee84c6248187646fe49551f31https://aws.amazon.com/blogs/machine-learning/detecting-and-recovering-from-silent-data-corruption-on-amazon-sagemaker-hyperpod/ Read discussion