GLM-5.3 on Two DGX Stations: NVIDIA Shows 753B Running Locally with vLLM

NVIDIA has published a step-by-step guide to serving Z.ai's 753-billion-parameter GLM-5.3 on two DGX Stations with vLLM. The model checkpoint is about 464 GB, and a single GB300 GPU holds 252 GB of HBM3e, so no one machine can run it. The fix is tensor parallelism across both Stations. In NVIDIA's own post, one user gets 107.4 tokens per second, and total throughput peaks at 607.8 tokens per second.

This is a local-AI recipe, not a product launch. It shows how far "frontier model on your own desk" has moved since we covered GLM-5.3's vision work and NVIDIA's eight-node DGX Spark cluster for the same model.

TL;DR: the setup in one table

Why does a 40B-active model still need 464 GB?

GLM-5.3 is a mixture-of-experts model. Only 40 billion of its 753 billion parameters run for any one token, which is why it decodes quickly. But a router picks different experts for each token, so every expert must stay resident in GPU memory. NVIDIA states this plainly: sparse activation does not shrink the memory requirement.

The NVFP4 format helps. It targets the linear weights and activations of the MoE blocks, which is why the checkpoint reports MIXED_PRECISION quantization instead of one uniform format. Even so, the checkpoint is 50 safetensors shards totalling about 464 GB.

High-bandwidth memory stack feeding a chip, illustrating why GLM-5.3 weights must fit in GPU memory

With tensor parallelism set to 2, each GPU holds roughly half of every weight tensor. NVIDIA's startup logs show about 218 GiB of weights per GPU. After loading and before any request, Station A used 231.35 GiB and Station B used 231.34 GiB. There is no CPU weight offload, so both GPUs compute every request and neither can answer alone.

How is the two-Station cluster wired?

Each DGX Station carries a GB300 GPU and ConnectX-8 networking. NVIDIA connects the two machines directly over two RoCE rails named cx8r0 and cx8r1. The worked example uses cx8r1, RDMA device mlx5_1, and the addresses 192.168.101.1 (Station A) and 192.168.101.2 (Station B).

Before starting, you check that the link reports 400000 Mb/s, that MTU is 9000, and that jumbo-frame pings succeed. Each Station needs its own full copy of the checkpoint, so you need at least 464 GB free on each, plus room for the container image. NVIDIA estimates the download at about an hour on a 1 Gb/s connection and about six minutes at 10 Gb/s. Station A copies the checkpoint to Station B with rsync over passwordless SSH.

The roles are asymmetric. Station A is rank 0 and hosts the OpenAI-compatible API on port 8000. Station B is rank 1 and runs a headless worker with no API server.

Which vLLM flags matter?

NVIDIA pins the container by SHA-256 digest so both ranks run identical software. The key choices are:

The NCCL detail is the one most likely to save you a day. NVIDIA uses NCCL 2.30.7 from the image, because older versions can deadlock on cross-node CUDA graph replays over RoCE (vLLM issue 52504). It also sets an InfiniBand timeout of 22, a retry count of 7 and GPUDirect RDMA level 5, pinned to the chosen device and GID index.

For background on how vLLM-style engines are changing for local hardware, see our coverage of NVIDIA Flare, a Rust local inference server.

How do you verify it works?

NVIDIA gives three checks. HEALTH_OK confirms Station A's API answers. RDMA_CHECK_OK confirms NCCL logs show GPUDirect RDMA with no socket fallback, warnings or errors. And nvidia-smi should show VLLM::Worker_TP0 on Station A and VLLM::Worker_TP1 on Station B.

The sample request asks a simple sales question (Q1 = 120, Q2 = 180, Q3 = 150) and returns a 50 percent Q1-to-Q2 increase with finish_reason: stop. The reasoning_effort parameter accepts low, high and max; higher settings allow longer reasoning at the cost of latency.

What did NVIDIA measure?

NVIDIA ran AIPerf 0.12.0 in its own virtual environment so it would not disturb vLLM's dependencies. The workload was 80 Coding conversations (89 turns) from SPEED-Bench, a set of real prompts for evaluating speculative decoding. Each concurrency level sends 3 times C warmup requests and 10 times C measured requests, at default (max) reasoning effort with an 8,192-token output cap. The full sweep takes several hours.

Two readings matter. First, peak total throughput arrives at 16 concurrent requests, and per-user speed drops from 115.3 tok/s at concurrency 1 to 42.4 tok/s at 16. Second, 32 concurrent requests is worse than 16 on every metric, which suggests the cluster is saturated, not scaling. These are NVIDIA's own figures on its own hardware, and the token counts include reasoning tokens. We have not reproduced them.

How does this compare with smaller local setups?

The two-Station recipe sits at the top of the local-AI range. Our DGX Spark local LLM guide covers the entry end, and NVIDIA's eight-node DGX Spark cluster showed the full GLM 5.3 at 31 tokens per second on 1 TB of pooled memory. Two Stations are far faster per user, but the hardware costs far more.

For lighter ways to run earlier GLM releases on modest machines, see Unsloth Studio with GLM-5.2 and the Colibri streaming approach.

What will readers ask next?

Can I run this on one Station? Not with this checkpoint. At 464 GB it exceeds the 252 GB of HBM3e on one GB300 GPU, and NVIDIA's recipe has no CPU weight offload. A smaller or more aggressively quantized model would be a different recipe.

Why not just use more concurrency? The table answers that. Going from 16 to 32 concurrent requests cut total throughput from 607.8 to 385.0 tokens per second and raised mean latency from 136 to 454 seconds. The configuration caps sequences at 16 for most runs for a reason.

Does speculative decoding help here? NVIDIA enables MTP with 4 draft tokens, which is part of why single-user inter-token latency is only 8.78 ms. The post does not publish a run with it turned off, so we cannot state the gain.

Where is the model card? NVIDIA publishes the checkpoint on Hugging Face as nvidia/GLM-5.3-NVFP4, under the NVIDIA Open Model Agreement plus the original GLM-5.3 license terms.

What are the risks and gotchas?

• Open API by default. The recipe binds --host 0.0.0.0 with host networking, which exposes an unauthenticated API on every interface of Station A. Add a firewall, authentication and TLS.
• Storage and time. Both machines need the full 464 GB. The benchmark sweep takes hours.
• Gated data. The benchmark dataset needs you to accept the gated cais/hle terms on Hugging Face with the same account.
• Licensing. The NVIDIA Open Model Agreement and the original GLM-5.3 license terms both apply.
• Memory pressure. If you hit CUDA out-of-memory, lower --max-model-len or concurrency.

What this means for what you build or pay

If you build agents, this is a reference for a private, no-API-bill endpoint with an OpenAI-compatible interface and tool-call parsing. At 107 tok/s for one user, a coding agent is usable. At peak load, the 608 tok/s total is enough for a small team, not a product.

If you pay for hosted inference, the cost question is hardware you own against tokens you rent. NVIDIA does not publish a price for this configuration in the post, so we cannot make that comparison. Treat the numbers as an upper bound for tutorial conditions, and test your own prompts before you commit.

The main lesson is architectural. Memory capacity, not compute, sets the floor for sparse models, and fast interconnect decides whether splitting across machines works. The same recipe scales down to smaller clusters.

Related reading

• GLM-5.3 vision and community input
• DGX Spark eight-node clusters and GLM 5.3
• NVIDIA Flare local inference server
• Best DGX Spark local LLM setup
• Surface RTX Spark Dev Box
• Unsloth Studio with GLM-5.2
• External: vLLM documentation and the model on Hugging Face

Specifications and benchmark numbers are accurate as of November 13, 2026, and come from NVIDIA's own post.