GLM-5.3 Hits 607.8 Tok/s on Two DGX Stations with vLLM

Officially launched on November 13, 2026.
NVIDIA has published a developer blog detailing how Zhipu AI's 753-billion-parameter GLM-5.3 NVFP4 model can be served from two linked DGX Station GB300 systems, achieving 607.8 tokens per second at peak throughput. Posted November 13, 2026, by NVIDIA engineer Fatima Fall, the benchmark demonstrates frontier-scale model deployment across multiple machines behind a single API endpoint using vLLM tensor parallelism and ConnectX-8 RDMA interconnect.
Deployment Architecture and Hardware Configuration
The 464GB GLM-5.3 checkpoint exceeds the memory capacity of any single GPU, requiring a distributed approach. Fall's configuration splits the model weights across two DGX Station GB300 units, each equipped with 252GB of HBM3e memory. The vLLM framework handles tensor parallelism (TP=2), allocating approximately 218 GiB of weights to each GPU while leaving headroom for activation memory and the KV-cache.
ConnectX-8 RDMA networking links the two stations, enabling low-latency cross-machine tensor communication. The setup uses FP8 precision for the key-value cache, prefix caching to reuse repeated prompt tokens, and four-token speculative decoding to accelerate generation. Maximum model length is capped at 98,304 tokens, balancing context window size with memory constraints.
Throughput Performance and Concurrency Scaling
Peak throughput of 607.8 tokens per second occurred at concurrency 16, with a time-to-first-token of 482.90 milliseconds and mean request latency of 136.46 seconds. However, the throughput curve is nonlinear. At concurrency 32, output dropped to 385.0 tok/s while mean latency ballooned to 453.5 seconds, indicating the system hits bottlenecks beyond the optimal load point.
Fall frames the result matter-of-factly: "You now have a 753-billion-parameter model running behind one local API on two NVIDIA DGX Stations, with every weight in GPU memory." The benchmark joins a series of NVIDIA posts tracking local deployment strategies for large-scale models, particularly Chinese open-source releases that often lack commercial cloud endpoints.
Release Date and Public Availability
The NVIDIA developer blog was officially published on November 13, 2026. The benchmark recipe and configuration details are publicly available, though the blog does not provide vLLM configuration files or installation scripts. GLM-5.3 itself is an open-source model from Zhipu AI, accessible for local deployment by organizations with sufficient hardware.
Implications for Frontier Model Serving
This benchmark addresses a practical constraint: as models cross the 500-billion-parameter threshold, single-machine inference becomes infeasible. The vLLM + RDMA approach offers a path to serve frontier models locally without resorting to model sharding across data centers or cloud infrastructure.
Fall's post is the latest in a 90-day sequence of NVIDIA deployment guides. For teams evaluating GLM-5.3 alternatives or comparing GLM-5.3 benchmarks across hardware configurations, the 607.8 tok/s figure provides a ceiling for dual-station setups. The nonlinear scaling behavior underscores the importance of tuning concurrency limits and batch sizes to avoid latency collapse.
What This Means
NVIDIA's benchmark confirms that two-machine tensor parallelism is viable for models approaching a trillion parameters, provided RDMA interconnect and careful vLLM tuning. Organizations with on-premises DGX infrastructure can now serve GLM-5.3 and similar open-weight models without offloading to third-party APIs. The 607.8 tok/s ceiling and concurrency-16 sweet spot give deployment teams concrete targets for capacity planning. As model sizes continue to grow, multi-station serving configurations may become the standard for local inference at the frontier.
on Emergent today
.avif)




