Oriel — AI & technology worth noticing, every day.

Build a Validated Kubernetes Runtime for NVIDIA Dynamo

Web page · https://developer.nvidia.com/blog/build-a-validated-kubernetes-runtime-for-nvidia-dynamo

Teams building AI applications need a repeatable path from a model configuration to a working inference endpoint. NVIDIA Dynamo is a distributed inference...

First seen by Oriel on .

AI summary

NVIDIA AI Cluster Runtime (AICR) packages and validates NVIDIA Dynamo with its Kubernetes infrastructure, providing reproducible deployments and two KV-aware routing options for inference workloads.

  • The Dynamo recipe pins component versions, compatibility constraints, placement settings, checksums, and validation expectations; it includes dynamo-platform 1.4.2 and supporting GPU, scheduling, health, metrics, and ingress components.
  • The default Dynamo Frontend router scores workers by KV-cache prefix overlap and load, then dispatches requests over TCP; it suits model-specific endpoints with a smaller topology.
  • The Gateway API/GAIE path uses an InferencePool and Dynamo EPP for KV-aware endpoint selection, while the Gateway handles transport; it suits shared Kubernetes ingress but needs additional components and worker sidecars.
  • Both paths use live worker-published KV-cache events; predicting cache state cannot detect evictions and may degrade routing quality under memory pressure.
  • AICR snapshots the cluster, generates a recipe, renders a Helmfile bundle, and validates the runtime before application teams deploy a DynamoGraphDeployment and test endpoints. The walkthrough’s tested versions target Amazon EKS with NVIDIA GB200 NVL72 and NVIDIA GB300 NVL72 systems.

#nvidia dynamo #kubernetes #inference serving #gateway api

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
NVIDIA Developer Bloghttps://developer.nvidia.com/blog/?p=124424https://developer.nvidia.com/blog/build-a-validated-kubernetes-runtime-for-nvidia-dynamo/ Read discussion