Oriel — AI & technology worth noticing, every day.

Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM

Web page · https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm

Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations...

First seen by Oriel on .

AI summary

This tutorial serves the 464 GB NVIDIA GLM-5.3 NVFP4 model across two NVIDIA DGX Stations using vLLM tensor parallelism and ConnectX-8 RDMA.

  • The MoE model has 753 billion parameters, activates 40 billion per token, and exceeds one GB300 GPU’s 252 GB HBM3e capacity.
  • Tensor parallelism at TP=2 splits weights and computation across both GPUs, while Station A exposes one OpenAI-compatible API and Station B runs headlessly.
  • Each Station stores the full pinned checkpoint, but vLLM loads about 218 GiB of weights per GPU without CPU weight offload.
  • The configuration supports 98,304-token contexts, FP8 KV cache, prefix caching, and four-token MTP speculative decoding, while allocating 93% of GPU memory.
  • The unauthenticated API listens on all Station A interfaces, so the tutorial recommends firewalling port 8000 and adding authentication and TLS before wider access.

#vllm #nvidia dgx #distributed inference #rdma

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 143, latest observation ; +138 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 69, latest observation ; +69 over observed period since

Aliases

SourceExternal IDURL
Hacker News50497155https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm/ Read discussion
NVIDIA Developer Bloghttps://developer.nvidia.com/blog/?p=124562https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm/ Read discussion