Oriel — AI & technology worth noticing, every day.

SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints

Web page · https://aws.amazon.com/blogs/machine-learning/sagemaker-inference-now-supports-multi-instance-gpu-to-maximize-gpu-utilization-for-model-endpoints

Amazon SageMaker Inference now supports GPU partitioning with NVIDIA Multi-Instance GPU (MIG), which divides one GPU into hardware-isolated partitions so you can run a different model in each or take a right-sized fraction of a newer accelerator. This post shows how two optional API fields place one model per partition, and what we measured against a whole older GPU.

First seen by Oriel on .

AI summary

Amazon SageMaker Inference adds NVIDIA MIG-based GPU partitioning, letting inference components share accelerators with hardware isolation and more precise capacity selection.

  • MIG provides hardware-isolated compute, memory, cache, and bandwidth partitions, enabling predictable latency, fault isolation, no cold starts, and GPU memory protection.
  • Two optional fields configure partitioning: AcceleratorPartitionLayouts on endpoint variants and AcceleratorPartitionType on inference components; existing invocation APIs remain unchanged.
  • Supported profiles cover P4d A100, P5 H100, G7 NVIDIA RTX PRO 4500 Blackwell, and G7e NVIDIA RTX PRO 6000 Blackwell Server Edition accelerators.
  • A partition layout applies uniformly across every accelerator, and each device supports at most seven partitions except G7 and G7e, which support two and four.
  • Workload sizing must account for compute-heavy prefill, bandwidth-bound decode, model weights, KV cache, concurrency, context length, and 1–2 GB runtime overhead.

#sagemaker #nvidia mig #gpu inference #model serving

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
AWS Machine Learning Blog45834970ac5a8ce39a39de22bbc2ae33f7618139https://aws.amazon.com/blogs/machine-learning/sagemaker-inference-now-supports-multi-instance-gpu-to-maximize-gpu-utilization-for-model-endpoints/ Read discussion