Oriel — AI & technology worth noticing, every day.

Memory-Enforced GPU Sharing in Kubernetes with NVIDIA CUDA 13.4

Web page · https://developer.nvidia.com/blog/memory-enforced-gpu-sharing-in-kubernetes-with-nvidia-cuda-13-4

Many AI workloads need only a fraction of a GPU. Giving each one an entire device can leave capacity idle; sharing GPUs lets platform teams serve more workloads...

First seen by Oriel on .

AI summary

CUDA 13.4 adds transparent GPU memory enforcement for shared workloads, while the open-source gpu-fractioning operator applies per-container memory and compute settings on Kubernetes nodes.

  • Workloads can request guaranteed GPU memory with an optional higher limit, and choose time-slicing or sm-sharing compute modes.
  • GPU Fractioning Operator runs MPS, configures workload limits from Pod annotations, and reports memory and compute use per fraction.
  • KAI Scheduler v0.18.0 integrates the operator through its NvFractions mode; other schedulers can use the feature by writing the same annotations.
  • Time-slicing runs workloads in turns with separate error domains; sm-sharing runs them concurrently but places them in a shared error domain.
  • Memory requests can be specified as absolute sizes or GPU portions, and can apply independently to multiple GPUs.

#kubernetes #gpu sharing #cuda #kai scheduler

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
NVIDIA Developer Bloghttps://developer.nvidia.com/blog/?p=124247https://developer.nvidia.com/blog/memory-enforced-gpu-sharing-in-kubernetes-with-nvidia-cuda-13-4/ Read discussion