NVIDIA Serves 753B-Parameter GLM-5.3 at 607.8 Tokens/Sec Across Two DGX Station GB300 Systems
NVIDIA demonstrates serving Zhipu AI 753B-parameter GLM-5.3 NVFP4 model at 607.8 tokens per second across two ConnectX-8 linked DGX Station GB300 systems using vLLM tensor parallelism.
NVIDIA has published a detailed engineering benchmark demonstrating how organizations can serve Zhipu AI 753-billion-parameter GLM-5.3 NVFP4 model entirely on local hardware at up to 607.8 tokens per second. By linking two NVIDIA DGX Station GB300 systems over ConnectX-8 RDMA and running vLLM tensor parallelism, the setup keeps all 464 GB of quantized model weights in HBM3e GPU memory behind a single OpenAI-compatible API endpoint.
Splitting 464 GB of Weights Across Dual 252 GB HBM3e GPUs
GLM-5.3 NVFP4 is a quantized Mixture-of-Experts model with 753 billion total parameters and 40 billion active parameters per token. Because its 464 GB checkpoint exceeds the 252 GB of HBM3e memory available on a single GB300 GPU, NVIDIA connected two DGX Station GB300 units via ConnectX-8 RDMA networking. Using vLLM tensor parallelism (TP=2), each GPU holds roughly 218 GiB of model weights while reserving ample memory headroom for activations and KV-cache.
vLLM Optimizations: FP8 KV-Cache and 4-Token Speculative Decoding
To maximize throughput across the dual-station cluster, the distributed vLLM runtime layers FP8 KV-cache quantization, prefix caching for repeated prompt blocks, and four-token speculative decoding (multi-token prediction) across a maximum context length of 98,304 tokens. This configuration allows enterprises and research labs to run frontier-scale open-weight models on-premises without cloud API dependencies.
AIPerf Coding Throughput: 607.8 Tok/s Peak at Concurrency 16
Evaluated with AIPerf on the SPEED-Bench Coding dataset, the dual-station setup achieved a peak overall output throughput of 607.8 tokens per second at concurrency 16, recording a 482.90 ms time-to-first-token and 136.46 seconds mean request latency. At concurrency 32, throughput tapered to 385.0 tokens per second as mean latency rose to 453.54 seconds, giving local deployment teams a concrete capacity-planning ceiling for multi-machine workstation inference.