Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM
Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations...
First seen by Oriel on .
AI summary
This tutorial serves the 464 GB NVIDIA GLM-5.3 NVFP4 model across two NVIDIA DGX Stations using vLLM tensor parallelism and ConnectX-8 RDMA.
- The MoE model has 753 billion parameters, activates 40 billion per token, and exceeds one GB300 GPU’s 252 GB HBM3e capacity.
- Tensor parallelism at TP=2 splits weights and computation across both GPUs, while Station A exposes one OpenAI-compatible API and Station B runs headlessly.
- Each Station stores the full pinned checkpoint, but vLLM loads about 218 GiB of weights per GPU without CPU weight offload.
- The configuration supports 98,304-token contexts, FP8 KV cache, prefix caching, and four-token MTP speculative decoding, while allocating 93% of GPU memory.
- The unauthenticated API listens on all Station A interfaces, so the tutorial recommends firewalling port 8000 and adding authentication and TLS before wider access.
#vllm #nvidia dgx #distributed inference #rdma
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 143, latest observation ; +138 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 69, latest observation ; +69 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50497155 | https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm/ Read discussion |
| NVIDIA Developer Blog | https://developer.nvidia.com/blog/?p=124562 | https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm/ Read discussion |