SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints
Amazon SageMaker Inference now supports GPU partitioning with NVIDIA Multi-Instance GPU (MIG), which divides one GPU into hardware-isolated partitions so you can run a different model in each or take a right-sized fraction of a newer accelerator. This post shows how two optional API fields place one model per partition, and what we measured against a whole older GPU.
First seen by Oriel on .
AI summary
Amazon SageMaker Inference adds NVIDIA MIG-based GPU partitioning, letting inference components share accelerators with hardware isolation and more precise capacity selection.
- MIG provides hardware-isolated compute, memory, cache, and bandwidth partitions, enabling predictable latency, fault isolation, no cold starts, and GPU memory protection.
- Two optional fields configure partitioning: AcceleratorPartitionLayouts on endpoint variants and AcceleratorPartitionType on inference components; existing invocation APIs remain unchanged.
- Supported profiles cover P4d A100, P5 H100, G7 NVIDIA RTX PRO 4500 Blackwell, and G7e NVIDIA RTX PRO 6000 Blackwell Server Edition accelerators.
- A partition layout applies uniformly across every accelerator, and each device supports at most seven partitions except G7 and G7e, which support two and four.
- Workload sizing must account for compute-heavy prefill, bandwidth-bound decode, model weights, KV cache, concurrency, context length, and 1–2 GB runtime overhead.
#sagemaker #nvidia mig #gpu inference #model serving
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| AWS Machine Learning Blog | 45834970ac5a8ce39a39de22bbc2ae33f7618139 | https://aws.amazon.com/blogs/machine-learning/sagemaker-inference-now-supports-multi-instance-gpu-to-maximize-gpu-utilization-for-model-endpoints/ Read discussion |