Oriel — AI & technology worth noticing, every day.

Configure, Certify, and Operate Kubernetes GPU Clusters with NVIDIA DSX OS

Web page · https://developer.nvidia.com/blog/configure-certify-and-operate-kubernetes-gpu-clusters-with-nvidia-dsx-os

One underperforming GPU can slow an entire distributed training job. The node may pass basic health checks and show normal utilization, leaving operators to...

First seen by Oriel on .

AI summary

NVIDIA DSX OS combines AICR, NVCRE, and NVSentinel to make Kubernetes GPU cluster configuration reproducible, validate workload readiness, and continuously manage health and recovery.

  • AICR generates version-pinned recipes and deployment artifacts for tools including Helm, Argo CD, Flux, and Helmfile; it can also validate live clusters and detect drift.
  • NVCRE runs NCCL, DCGM, and NVIDIA NeMo workload tests, records certification results, and bisects failing groups to identify suspect nodes.
  • Certification thresholds are optional; without them, performance measurements are reported but below-target nodes can still pass.
  • NVSentinel monitors existing health signals without consuming GPU time and can quarantine, drain, remediate, and trigger revalidation when configured.
  • NVSentinel defaults to monitoring mode; operators can enable lifecycle validation for new or repaired nodes, while upgrades and fabric changes require an explicit validation request or pipeline trigger.

#kubernetes #gpu clusters #cluster monitoring #nvidia nccl

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
NVIDIA Developer Bloghttps://developer.nvidia.com/blog/?p=124575https://developer.nvidia.com/blog/configure-certify-and-operate-kubernetes-gpu-clusters-with-nvidia-dsx-os/ Read discussion