HomeNews

GLM-5.3 Hits 607.8 Tok/s on Two DGX Stations with vLLM

Swapnil Palash
Swapnil Palash
•
Nov 15, 2026 3:38 AM
•
0
 min read
Select Emergent as your Preferred news source
GLM-5.3 Hits 607.8 Tok/s on Two DGX Stations with vLLM

Officially launched on November 13, 2026.

💡 TL;DR

  • NVIDIA benchmarked Zhipu's 753-billion-parameter GLM-5.3 NVFP4 model distributed across two DGX Station GB300 systems linked by ConnectX-8 RDMA.
  • The vLLM tensor-parallel deployment achieved 607.8 tokens per second at concurrency 16 with 482ms time-to-first-token before degrading at higher loads.
  • The 464GB checkpoint used FP8 KV-cache, prefix caching, and four-token speculative decoding across 252GB HBM3e GPUs in each station.

NVIDIA has published a developer blog detailing how Zhipu AI's 753-billion-parameter GLM-5.3 NVFP4 model can be served from two linked DGX Station GB300 systems, achieving 607.8 tokens per second at peak throughput. Posted November 13, 2026, by NVIDIA engineer Fatima Fall, the benchmark demonstrates frontier-scale model deployment across multiple machines behind a single API endpoint using vLLM tensor parallelism and ConnectX-8 RDMA interconnect.

Deployment Architecture and Hardware Configuration

The 464GB GLM-5.3 checkpoint exceeds the memory capacity of any single GPU, requiring a distributed approach. Fall's configuration splits the model weights across two DGX Station GB300 units, each equipped with 252GB of HBM3e memory. The vLLM framework handles tensor parallelism (TP=2), allocating approximately 218 GiB of weights to each GPU while leaving headroom for activation memory and the KV-cache.

ConnectX-8 RDMA networking links the two stations, enabling low-latency cross-machine tensor communication. The setup uses FP8 precision for the key-value cache, prefix caching to reuse repeated prompt tokens, and four-token speculative decoding to accelerate generation. Maximum model length is capped at 98,304 tokens, balancing context window size with memory constraints.

Throughput Performance and Concurrency Scaling

Peak throughput of 607.8 tokens per second occurred at concurrency 16, with a time-to-first-token of 482.90 milliseconds and mean request latency of 136.46 seconds. However, the throughput curve is nonlinear. At concurrency 32, output dropped to 385.0 tok/s while mean latency ballooned to 453.5 seconds, indicating the system hits bottlenecks beyond the optimal load point.

Fall frames the result matter-of-factly: "You now have a 753-billion-parameter model running behind one local API on two NVIDIA DGX Stations, with every weight in GPU memory." The benchmark joins a series of NVIDIA posts tracking local deployment strategies for large-scale models, particularly Chinese open-source releases that often lack commercial cloud endpoints.

Release Date and Public Availability

The NVIDIA developer blog was officially published on November 13, 2026. The benchmark recipe and configuration details are publicly available, though the blog does not provide vLLM configuration files or installation scripts. GLM-5.3 itself is an open-source model from Zhipu AI, accessible for local deployment by organizations with sufficient hardware.

Implications for Frontier Model Serving

This benchmark addresses a practical constraint: as models cross the 500-billion-parameter threshold, single-machine inference becomes infeasible. The vLLM + RDMA approach offers a path to serve frontier models locally without resorting to model sharding across data centers or cloud infrastructure.

Fall's post is the latest in a 90-day sequence of NVIDIA deployment guides. For teams evaluating GLM-5.3 alternatives or comparing GLM-5.3 benchmarks across hardware configurations, the 607.8 tok/s figure provides a ceiling for dual-station setups. The nonlinear scaling behavior underscores the importance of tuning concurrency limits and batch sizes to avoid latency collapse.

What This Means

NVIDIA's benchmark confirms that two-machine tensor parallelism is viable for models approaching a trillion parameters, provided RDMA interconnect and careful vLLM tuning. Organizations with on-premises DGX infrastructure can now serve GLM-5.3 and similar open-weight models without offloading to third-party APIs. The 607.8 tok/s ceiling and concurrency-16 sweet spot give deployment teams concrete targets for capacity planning. As model sizes continue to grow, multi-station serving configurations may become the standard for local inference at the frontier.

About the writer

Swapnil is an Engineer at Emergent, where he builds the agent stack — orchestration, memory, and sandboxed execution for agents that act on your behalf. He was previously a Staff Engineer at LinkedIn, worked on Google Spanner and holds a Computer Science degree from IIT Delhi.

Start Building
on Emergent today
Try Emergent