Oriel — AI & technology worth noticing, every day.

Diagnose Distributed AI Workload Failures with NVIDIA Attribution

Web page · https://developer.nvidia.com/blog/diagnose-distributed-ai-workload-failures-with-nvidia-attribution

In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state...

First seen by Oriel on .

AI summary

NVIDIA Attribution is an open source, workload-centric system that correlates distributed AI telemetry with deterministic rules to distinguish root causes from downstream symptoms and return structured, reproducible failure diagnoses.

  • NVIDIA Attribution analyzes millions of workloads weekly across hundreds of thousands of GPUs and has been used across NVIDIA AI infrastructure for years.
  • The Blueprint collects scheduler context, logs, and metrics; the Service applies the attribution engine through an HTTP API and returns one verdict per workload.
  • In the example, XID 149 on node-02 identifies an NVLink signal integrity issue, while subsequent NCCL errors are retained as supporting evidence.
  • Diagnoses include terminal_state, sub_domain, affected nodes, symptoms, evidence, collector coverage, and errors; incomplete telemetry can produce an ambiguous sub_domain.
  • The Blueprint supports Slurm, Kueue, Loki, Kubernetes logs, PromQL, SQLite, and local inputs, while developers can add connectors and tested declarative or Python rules.

#nvidia attribution #distributed ai #gpu reliability #failure diagnosis

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
NVIDIA Developer Bloghttps://developer.nvidia.com/blog/?p=124515https://developer.nvidia.com/blog/diagnose-distributed-ai-workload-failures-with-nvidia-attribution/ Read discussion