NVIDIA Serves GLM-5.3 Across Two DGX Stations at 607.8 tok/s with vLLM
By Mr.Xu Community Post
Published:
Summary:NVIDIA demonstrates serving GLM-5.3 with vLLM across two NVIDIA DGX Stations, reporting throughput of 607.8 tokens per second. The deployment illustrates how tensor parallelism, continuous batching, and efficient KV-cache management can scale large-model serving beyond a single workstation. However, the supplied excerpt does not disclose prompt and output lengths, concurrency, time-to-first-token, quantization, or the full hardware configuration, so the figure should be treated as a workload-spe
GLM-5.3 Runs Across Two DGX Stations at 607.8 tok/s with vLLM
NVIDIA has demonstrated serving GLM-5.3 with vLLM across two DGX Stations, reporting aggregate throughput of 607.8 tokens per second. The main engineering value is not simply the headline number, but the use of multiple workstations to serve a large model that would otherwise strain a single system. vLLM's continuous batching can admit new requests as others finish, while paged KV-cache management reduces memory fragmentation caused by statically reserving cache space.
How Multi-Node Tensor Parallelism Works
When a model's weights do not fit on one workstation, vLLM can distribute them using tensor parallelism. If a weight matrix (W) is split column-wise into (W_1,\ldots,W_k), the output is computed as:
[ y = \sum_{i=1}^{k} xW_i ]
Each GPU computes a partial output, followed by an All-Reduce or Reduce-Scatter operation across the high-speed interconnect. This allows a larger model to fit in aggregate memory and improves batch throughput, but every decoding step may require cross-node synchronization.
The trade-offs are substantial:
- Communication overhead: More aggressive tensor-parallel partitioning increases collective operations and exposes the service to interconnect bottlenecks.
- Memory pressure: GPU memory must hold weights, activations, runtime buffers, and the KV cache.
- Larger failure domain: A slow or failed node can affect the entire model replica.
- Non-linear scaling: A second workstation may not double throughput because communication and serial execution remain on the critical path.
Throughput Alone Does Not Describe the Service
For an individual request, generation speed is approximately:
[ R_{\text{per-user}} = \frac{N_{\text{output}}}{T_{\text{TTFT}}+T_{\text{decode}}} ]
where (T_{\text{TTFT}}) is time to first token and (T_{\text{decode}}) covers subsequent token generation. Aggregate service throughput can be much higher because many requests advance concurrently. The 607.8 tok/s figure therefore requires context: it may represent aggregate output throughput rather than the speed perceived by one user.
Production evaluation should also track:
- Time to first token, particularly for long prompts dominated by prefill;
- Per-request decode speed and end-to-end P95/P99 latency;
- Input-to-output token ratios;
- KV-cache utilization and maximum sustainable concurrency;
- GPU utilization across batch sizes.
KV-cache memory can be approximated as:
[ M_{\text{KV}} \approx 2 \times L \times H_{\text{kv}} \times N_{\text{tokens}} \times S ]
where (L) is the number of layers, (H_{\text{kv}}) is the aggregate K/V head dimension, (N_{\text{tokens}}) is the number of cached tokens, and (S) is bytes per element. Grouped-query attention or MLA-style compression can reduce the cache footprint, but the actual benefit depends on the model's implementation rather than the serving framework alone.
Interpreting NVIDIA's Reported Result
The deployment blog provides valuable first-party configuration information, but it remains a vendor-reported test. A rigorous comparison requires the full hardware topology, tensor-parallel degree, quantization format, kernel configuration, batch size, prompt length, output length, and latency percentiles. It is also necessary to clarify whether the reported rate counts input tokens, output tokens, or both, and whether it includes visual encoding, networking, and scheduling overhead.
The headline throughput should not be read as a universal property of GLM-5.3 or vLLM. A short-input benchmark can yield far more attractive aggregate numbers than a production workload with long prompts and variable generation lengths.
Deployment Guidance for Developers
Teams planning a similar multi-workstation setup should:
- Define service-level objectives first: establish TTFT, per-user token rate, concurrency targets, and P99 latency.
- Validate the interconnect: bandwidth, latency, and NUMA placement can dominate multi-node performance.
- Stress-test KV-cache behavior: measure preemption, cache-hit rate, and fragmentation under mixed prompt lengths.
- Compare parallelism strategies: tensor, pipeline, and data parallelism should be evaluated rather than assuming one scaling path.
- Design for node failure: include model reload, traffic draining, and degraded-mode procedures.
- Replay real workloads: use production-like prompt and output distributions instead of a fixed synthetic generation length.
vLLM supplies the batching, caching, and distributed-execution primitives needed for this kind of deployment. NVIDIA's case demonstrates the feasibility of serving GLM-5.3 across two DGX Stations and supplies a useful throughput reference, but production claims still require independent latency, cost, and scaling-efficiency measurements.
— END —Source: Hacker News AI Feed (2026-11-14)
Tags: #NVIDIA #DGX Station #vLLM #GLM-5.3 #Tensor Parallelism
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments