Accelerating Reinforcement Learning Weight Refit with the Reshard Collective
During reinforcement learning (RL) post-training, training trajectories are generated on the fly by running inference with the current policy model. For...
First seen by Oriel on .
AI summary
This NVIDIA post introduces NCCL M2N Reshard, a DTensor-based collective that reduces reinforcement-learning weight-refit communication and delivers 2.7x–5.7x lower latency than gather-and-broadcast.
- Conventional refit all-gathers complete parameters and broadcasts them, producing approximately twice the model weight size in logical traffic, or about 10 TB for cited models.
- Reshard computes intersections between source and destination shards, transferring only required regions while coordinating network traffic and NVLink exchanges across replica groups.
- NVIDIA NeMo RL uses a hybrid design: feed-forward projection weights, representing around 95% of parameters, use Reshard while smaller weights retain gather-and-broadcast.
- Benchmarks on NVIDIA DGX H100 and NVIDIA DGX GB200 show refit latencies from 1.9 to 2.6 seconds with Reshard, versus 6.1 to 14.9 seconds conventionally.
- A refit-only microbenchmark completed Qwen3.8 2.4T BF16 in 4.3 seconds; NeMo RL currently recreates communication groups when GPU membership changes.
#reinforcement learning #nccl #dtensor #model parallelism
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| NVIDIA Developer Blog | https://developer.nvidia.com/blog/?p=124653 | https://developer.nvidia.com/blog/accelerating-reinforcement-learning-weight-refit-with-the-reshard-collective/ Read discussion |