Diagnose Distributed AI Workload Failures with NVIDIA Attribution
Web page · https://developer.nvidia.com/blog/diagnose-distributed-ai-workload-failures-with-nvidia-attribution
In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state...
First seen by Oriel on .
AI summary
NVIDIA Attribution is an open source, workload-centric system that correlates distributed AI telemetry with deterministic rules to distinguish root causes from downstream symptoms and return structured, reproducible failure diagnoses.
- NVIDIA Attribution analyzes millions of workloads weekly across hundreds of thousands of GPUs and has been used across NVIDIA AI infrastructure for years.
- The Blueprint collects scheduler context, logs, and metrics; the Service applies the attribution engine through an HTTP API and returns one verdict per workload.
- In the example, XID 149 on node-02 identifies an NVLink signal integrity issue, while subsequent NCCL errors are retained as supporting evidence.
- Diagnoses include terminal_state, sub_domain, affected nodes, symptoms, evidence, collector coverage, and errors; incomplete telemetry can produce an ambiguous sub_domain.
- The Blueprint supports Slurm, Kueue, Loki, Kubernetes logs, PromQL, SQLite, and local inputs, while developers can add connectors and tested declarative or Python rules.
#nvidia attribution #distributed ai #gpu reliability #failure diagnosis
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| NVIDIA Developer Blog | https://developer.nvidia.com/blog/?p=124515 | https://developer.nvidia.com/blog/diagnose-distributed-ai-workload-failures-with-nvidia-attribution/ Read discussion |