Serving a 753B GLM-5.3 Model Across Two NVIDIA DGX Stations with vLLM and Tensor Parallelism

This guide shows how to serve NVIDIA’s GLM-5.3 NVFP4 model across two GB300 NVIDIA DGX Stations using vLLM and tensor parallelism (TP=2). GLM-5.3 is a quantized mixture-of-experts (MoE) model with 753 billion parameters and about 40 billion activated per token; its checkpoint is roughly 464 GB, larger than a single GB300’s 252 GB HBM3e, so two stations are used to keep all weights in GPU memory.

Why this matters

Frontier-scale models often no longer fit on a single GPU or host. Splitting tensors across GPUs and using RDMA (ConnectX‑8) lets you expose a single OpenAI-compatible endpoint that serves the full model locally — useful for private-data inference, local agent prototyping, and avoiding per-token hosted costs.

Prerequisites

• Two GB300 NVIDIA DGX Stations running Linux, connected and validated with the "Connect Two DGX Stations playbook", including the GPU-memory RDMA tests.
• SSH and sudo access on both stations with the same Linux username. Docker and the NVIDIA Container Toolkit are provided in the DGX Station stack.
• Outbound access to container registries, Hugging Face and package repositories.
• At least ~464 GB free disk on each station for the model checkpoint plus room for the vLLM image and temporary files.
• Acceptance of the NVIDIA Open Model Agreement and GLM-5.3 license terms; SPEED‑Bench is governed by the NVIDIA Evaluation Dataset License Agreement.
• A Hugging Face account and read token with access to the gated cais/hle dataset to fetch SPEED‑Bench prompts.

Workflow overview

1. Prepare both stations: identify the RDMA rail (for example cx8r1) and RDMA device (for example mlx5_1), create ~/glm53.env on each station with RANK (0 or 1), HEAD_IP, WORKER_IP, RAIL_IFACE, RAIL_HCA and other values.
2. Pull the pinned vLLM Docker image on both nodes, download the GLM-5.3-NVFP4 checkpoint on Station A (~464 GB) and rsync it to Station B. Verify file lists, counts (50 model shards), and config (architectures == ["GlmMoeDsaForCausalLM"] and quantization_config.quant_algo == "MIXED_PRECISION").
3. Launch the distributed vLLM service on both stations with the provided launch script (~/launch-glm53.sh). Important options include --nnodes 2, --node-rank, --tensor-parallel-size 2, --max-model-len 98304, and --gpu-memory-utilization 0.93. The script configures NCCL/RoCE, FP8 KV-cache, speculative decoding (MTP with 4 draft tokens), and prefix caching. Station A (RANK=0) runs the API; Station B runs a headless worker.
4. Verify operation: check the /health endpoint on Station A, confirm NCCL logs show GPU Direct RDMA and no fallback to sockets, and observe that each GPU holds roughly half the weights (startup reports ~218 GiB of weights per GPU). Example chat completion demonstrates a working OpenAI-compatible endpoint.
5. Benchmark concurrency using AIPerf and SPEED‑Bench coding prompts. The tutorial runs concurrency levels C=1, 8, 16 and 32 (3×C warmups, 10×C measured requests), collects server metrics and raw responses, and reports throughput and latency.

Checkpoint and memory details

• Checkpoint size: ~464 GB; both disks keep complete copies. vLLM partitions tensors across the two GB300 GPUs at load time (no further quantization).
• Because GLM-5.3 is MoE, only a subset of experts activate per token (~40 billion parameters activated), but all weights must be available since routing can target different experts.
• FP8 KV-cache reduces cache memory compared to 16-bit, allowing larger contexts.
• Each token’s processing involves both ranks: neither GPU holds a full model copy, and they exchange partial results over the RoCE rail.

Representative verification and example request

• Health check on Station A: curl to http://127.0.0.1:8000/health should return HEALTH_OK.
• NCCL logs should contain indications of GPU Direct RDMA (GDRDMA) and report the selected RDMA HCA; nvidia-smi should show VLLM::Worker_TP0 on Station A and VLLM::Worker_TP1 on Station B.
• Startup should report ~218 GiB of weights per GPU (about half the checkpoint), confirming tensor parallel partitioning.
• Example chat request to /v1/chat/completions returns a short completion and finish_reason: stop.

Example expected reply for the sample prompt in the tutorial (Q1=120; Q2=180; Q3=150):

• Trend: Units sold rose sharply in Q2 before partially declining in Q3, remaining above the Q1 level.
• Q1 → Q2 percentage change: 50% increase

Benchmark results (SPEED‑Bench Coding, prefix caching enabled)

Measured results published in the tutorial for concurrency settings:

• C=1: overall throughput 107.4 tok/s; mean request latency 49.84 s; mean TTFT 479.75 ms; mean inter-token latency 8.78 ms/token
• C=8: overall throughput 386.0 tok/s; mean request latency 114.17 s; mean TTFT 433.79 ms; mean inter-token latency 19.01 ms/token
• C=16: overall throughput 607.8 tok/s (peak); mean request latency 136.46 s; mean TTFT 482.90 ms; mean inter-token latency 23.77 ms/token
• C=32: overall throughput 385.0 tok/s; mean request latency 453.54 s; mean TTFT 899.36 ms; mean inter-token latency 77.90 ms/token

Overall throughput peaks at C=16 (607.8 tok/s) and drops at C=32, where average request latency becomes very large (453.5 s). Mean per-user throughput also declines as concurrency increases.

Running the benchmark

• Install AIPerf in a dedicated virtualenv on Station A to avoid contaminating the serving container.
• Prepare SPEED‑Bench coding prompts (the tutorial downloads and prepares 80 coding conversations — 89 turns total — from gated datasets; ensure you have accepted access to cais/hle on Hugging Face).
• Run a sweep across desired concurrencies; the tutorial provides an example loop to run C=1 and C=8 for a quick test and explains how to run a full sweep.

Troubleshooting

• Missing environment vars or SSH auth: source ~/glm53.env and validate key-based SSH from A to B.
• Gated dataset errors: sign in to Hugging Face, open cais/hle and select "Agree and access repository".
• Startup or RDMA failures: check docker logs on both nodes, validate checkpoint integrity and RoCE GID index, and rerun the playbook’s validation scripts.
• CUDA OOM: reduce --max-model-len or reduce concurrency.

Shutdown and restart

• Stop the vLLM container on A then B: docker stop --timeout 60 "$CONTAINER" and docker rm "$CONTAINER".
• Verify containers exited and no vLLM processes remain in nvidia-smi. Checkpoint files and benchmark results remain on disk; to resume, start containers in reverse order (B then A) and re-run the health and RDMA checks.

Practical outcome

Following these steps yields a local service exposing a single API that serves a 753‑billion‑parameter GLM-5.3 model across two GB300 DGX Stations with all weights resident in GPU memory. This setup enables private inference, local agent prototyping, and controlled performance tuning with realistic concurrency benchmarks.

Notes

• The tutorial’s launch script exposes the API on 0.0.0.0 with --network host; in production, add firewalling, authentication and TLS.
• The guide includes exact environment and launcher snippets, verification commands, and benchmarking scripts to reproduce the reported behaviour and measurements.