Grace Blackwell needs a new scheduler: 27% more usable GPUs from the same racks
Web page · https://skypilot.ai/blog/grace-blackwell-needs-a-new-scheduler
First seen by Oriel on .
AI summary
The article argues that NVL72 clusters need rack-aware scheduling and failure recovery to realize NVLink performance, reporting 27% more usable capacity with 16-node slices and 14.4% higher training throughput from rack-local placement.
- Tensor and expert parallelism rely on communication-intensive all-reduce and all-to-all operations; moving training off NVLink can reduce throughput by 43%.
- NVL72 connects 72 GPUs through a non-blocking rack fabric, providing 1.8 TB/s per GPU and 130 TB/s aggregate internal bandwidth.
- With roughly 2% node unavailability, a modeled 128-rack cluster provides 2,037 usable nodes with 16-node slices, versus 1,602 with full-rack slices.
- SkyPilot’s Topology-Aware Scheduler keeps workers within racks, while Frontier Trainer replaces failed nodes with reserved spares and Automatic Defragmentation frees racks for blocked jobs.
- In NVIDIA GR00T N1.7 pre-training across 512 GB300 GPUs, rack-aware placement cut iteration time from 1.59 s to 1.39 s; Auto Defrag only preempts workloads that can safely resume from checkpoints unless an admin opts in others.
#nvl72 #gpu scheduling #distributed training #nvlink
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 144, latest observation ; +137 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 22, latest observation ; +22 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50290912 | https://skypilot.ai/blog/grace-blackwell-needs-a-new-scheduler Read discussion |