NVIDIA enables 1 TB unified memory with eight-node DGX Spark clusters for local frontier‑class models

At GTC Berlin, NVIDIA announced official support for eight-node DGX Spark clusters. The change raises unified memory capacity from 128 GB per DGX Spark node—reaching 512 GB with four nodes—to 1 TB with eight nodes, enabling local execution of frontier-class models, multi-agent workloads and near-linear scaling for some pre-training tasks.

What eight nodes enable

• Memory: each DGX Spark node provides 128 GB of unified memory. Four nodes yield 512 GB; eight nodes yield 1 TB of unified memory, sufficient to load models that previously would not start on a four-node system. NVIDIA cites the full GLM 5.3 (753B) checkpoint from Z.ai as an example that runs across an eight-node cluster.

• Interactive serving: the cluster is designed for interactive use, not just batch loading. NVIDIA reported that GLM 5.3 Flash run across an eight-node cluster produced roughly 80 tokens per second (TPS) in generation, which is suitable for local interactive development against a frontier‑class mixture‑of‑experts model.

Performance data (full GLM 5.3, nvidia/GLM-5.3-NVFP4, ~470 GB NVFP4 checkpoint, measured with MTP3 and 8K input sequence length):

• Examples of concurrency vs. throughput:
  • 1 user: prefill 1,600 TPS (total), decode 31 TPS (total)
  • 4 users: prefill 1,600 TPS (total), prefill per user 750 TPS, decode total 51 TPS
  • 8 users: prefill 1,600 TPS (total), prefill per user 473 TPS, decode total 67 TPS

Faster collectives with a Spark‑optimized backend (RoCEnante)

Tensor‑parallel inference spends a substantial portion of time in collective operations such as all‑reduce. NVIDIA developed a Spark‑optimized collective backend called RoCEnante for switch‑connected clusters; it targets the low‑concurrency, small‑message, modest‑rank patterns typical of tensor‑parallel decode on local AI hardware.

RoCEnante performs an all‑to‑all exchange followed by local reduction: each rank shares its input with every other rank and then reduces the inputs locally, with a single GPU kernel coordinating exchange and reduction. This reduces the number of communication steps compared with ring all‑reduce, which forwards data through neighboring ranks over multiple rounds. The tradeoff is increased data movement, since each rank sends its full input to all peers; as message sizes or rank counts grow, that extra traffic becomes costly and ring all‑reduce’s bandwidth efficiency becomes more attractive.

Measured benefits:

• At four nodes (NVFP4), roughly 5% higher prefill and 21% higher decode throughput were observed.
• At eight nodes (non‑quantized model), around 18% higher decode throughput was measured.

RoCEnante is merged into FlashInfer and falls back to NCCL transparently if its startup test fails. It only applies to switch‑connected topologies; NCCL remains the supported default and better choice for larger clusters or high concurrency.

Cluster scaling from two to eight nodes

With low‑latency NVIDIA ConnectX‑7 NICs and RoCE, DGX Spark nodes exchange tensors efficiently, allowing models to be spread across more nodes to fit in memory and to improve throughput. NVIDIA benchmarked a dense Qwen3.8‑27B (NVFP4) model without speculative decoding across one to eight nodes (llama‑benchy, 8192 ISL, 256 OSL, concurrency 1):

• 1 node: prefill 2,635 TPS, decode 11.86 TPS (baseline 1.00x)
• 2 nodes (direct connect): prefill 3,285 TPS (1.25x), decode 21.71 TPS (1.84x)
• 4 nodes (physical ring): prefill 4,446 TPS (1.69x), decode 32.82 TPS (2.77x)
• 4 nodes (switch + RoCEnante): prefill 4,556 TPS (1.73x), decode 36.34 TPS (3.06x)
• 8 nodes (physical ring): prefill 4,962 TPS (1.88x), decode 40.11 TPS (3.38x)
• 8 nodes (switch + RoCEnante): prefill 5,105 TPS (1.94x), decode 52.58 TPS (4.43x)

Two patterns emerge: decode throughput scales strongly with node count (roughly 12 TPS on one node to 52 TPS on eight), and the best topology depends on workload. A physical ring works well when a single model is distributed across all nodes; a switch is required to split the cluster into independent groups or to support any‑to‑any connectivity.

Switchless ring topology and setup simplicity

Not every deployment needs a switch. DGX Spark nodes can be wired directly in a ring, reducing cost and simplifying setup—an attractive option for enthusiasts and small deployments. NCCL defaults to ring‑based algorithms in a four‑node deployment, which aligns with traffic flows for tensor‑parallel communication and gives good throughput for that configuration.

However, requirements change as you scale or specialize:

• To split an eight‑node cluster into two groups of four, a ring cannot express that topology and a switch is needed.
• Workloads that require point‑to‑point patterns (e.g., MPI‑based) need a switch to avoid hop penalties.
• The RoCEnante backend requires direct point‑to‑point connectivity between every node and therefore only applies to switched configurations.

In short: choose a switchless ring for the simplest, lowest‑cost four‑node inference box; choose a switch if you need eight nodes, split allocations, or communication patterns a ring can’t serve.

Multi‑model orchestration with NeMo Switchyard

Not every request requires the largest available model. On a shared eight‑node cluster, routing easy requests to a small fast model and reserving a larger model for hard requests can substantially increase aggregate throughput.

NVIDIA NeMo Switchyard implements a lightweight classifier that inspects each request or session and routes it to appropriate model pools, with a load balancer distributing traffic across replicas. Switchyard runs on switch‑connected clusters because any‑to‑any connectivity is required to route requests across replicas.

Prototype example:

• Classifier/execution model for simple tasks: OpenBMB’s MiniCPM5 2B
• Execution model for complex tasks: Qwen3.8 Flash Next
• On a standard‑plus‑hard tool‑use benchmark, the routed agent matched the Qwen‑only configuration in quality (91/100 on a 92‑case benchmark) while freeing cluster capacity.

Because MiniCPM serves roughly an order of magnitude more requests per second per replica than Qwen, routing a portion of traffic to the smaller model raises the cluster’s throughput ceiling. In one allocation across eight nodes (three Qwen replicas taking two Spark nodes each, two MiniCPM replicas taking one Spark node each), NVIDIA reports approximately:

• ~1.5x aggregate throughput vs. a Qwen‑only baseline when half of requests go to Qwen
• ~3.0x aggregate throughput when a quarter of requests go to Qwen and the rest to MiniCPM

Routing policies are simple YAML files; NVIDIA provides a sample policy as a starting point.

Pre‑training from scratch: near‑linear scaling with nanochat

Training scales differently from inference. When each node keeps a full copy of the model, only gradient synchronization is needed across nodes once per step, so clustering can scale nearly linearly. NVIDIA pre‑trained a nanochat model from scratch (~896.5M parameters, 20 transformer layers) and measured training throughput as nodes were added (global batch 524,288 tokens per step, 32 sequences per GPU microbatch):

• 1 node: 8,741 TPS (1.00x, 100.0% efficiency)
• 2 nodes: 17,202 TPS (1.97x, 98.4% efficiency)
• 4 nodes: 33,640 TPS (3.85x, 96.2% efficiency)
• 8 nodes: 68,079 TPS (7.79x, 97.4% efficiency)

Eight nodes delivered a 7.79x speedup at 97.4% efficiency, indicating that an eight‑node DGX Spark cluster is a credible platform for training small models from scratch, not only for serving very large ones. NVIDIA has published code to reproduce the nanochat experiment on GitHub.

Getting started with multi‑system DGX Spark

Eight‑node DGX Spark clusters provide the memory to load frontier‑class models, the flexibility to orchestrate multiple models concurrently, and the efficiency to pre‑train small models locally. NVIDIA also announced that NVIDIA Sync’s Cluster Assistant will handle setup and device management, with eight‑node support in NVIDIA Sync expected by the end of the year.

Resources and guidance NVIDIA highlights include connecting multiple DGX Spark machines for distributed workloads, serving LLMs with vLLM across DGX Spark, and model compatibility and Switchyard orchestration guides. Developers can start building on NVIDIA DGX Spark to run larger models, route workloads across model pools, and pre‑train locally before scaling out.