When to choose dense vs. Mixture-of-Experts (MoE) model architectures

There are two common ways to organize parameters in large language models: dense (every parameter participates for every token) and Mixture-of-Experts (MoE, where each token is routed to only a subset of experts). How a model structures its parameters influences throughput, memory cost, and serving complexity often more than the raw parameter count.

Basic difference

• Dense model: on each forward pass every parameter is active. For example, in a 27B dense model all 27 billion parameters contribute via a single shared feed-forward (FFN) block per decoder layer.
• MoE model: a decoder layer contains multiple FFN blocks (experts) and a router network chooses the top-k experts for each token. Only the selected FFN blocks run for that token; the others are skipped. Many modern MoE models also include a shared expert that receives all tokens.

How routing works

Routing is decided independently for each decoder layer, so a token is rerouted at each layer based on its intermediate representation. Experts tend to specialize more in syntax and token-type patterns (punctuation, numbers, etc.) than in broad topical domains, though this depends on architecture and training.

The router determines which FFN blocks are skipped, but tokens still pass through the full attention and embedding layers. When a model advertises "3B active parameters," that figure typically includes attention and embedding weights plus the selected FFN weights for the token.

Variants and hybrids

Some models use hybrid designs. NVIDIA’s Nemotron 3.5 Lightning is described as a Mamba-2 + MoE + Attention hybrid: Mamba-2 layers replace attention in many layers and carry a fixed-size recurrent state rather than an expanding KV cache. Lightning also computes routing decisions in a compressed space to make routing cheaper. MoE is a form of sparsity but the details matter: routing, recurrent projections, and other components affect behavior.

Which is faster: dense or MoE?

MoE models often achieve higher token throughput because only a subset of FFN parameters are activated per token. Dense models run the full network but offer simpler serving and more predictable latency.

At high concurrency, routing overhead and memory movement can reduce MoE’s advantage. Results depend on hardware, precision, inference framework, and model-specific design (for example, Nemotron 3.5 Lightning’s Mamba-2 layers and speculative decoding contribute beyond the MoE design itself).

A key point: MoE decouples memory from compute. Dense hosting and inference costs scale together, while MoE requires loading all experts into memory (fixed memory cost) and incurs compute cost only for the experts that fire (variable per token). Idle experts still consume memory but not compute. This trade-off is the central difference between the approaches.

For batch size 1, decoding is commonly memory-bound, which favors MoE because fewer weight bytes must be read per token. As batch size increases, tokens together tend to trigger more of the model’s experts, narrowing MoE’s relative advantage on throughput, although the per-token reduction in work remains. Most modern frameworks keep all experts in GPU memory simultaneously, which can reduce available KV cache space compared to a similarly sized dense model.

Example comparison (benchmarks and numbers)

Artificial Analysis provided median throughput and cost figures (10K-token input; retrieved Aug 31, 2026) for several models around the 30B parameter scale. Selected entries:

• Gemma 4 31B (dense, multimodal): total params 31B, active params 31B, ~61 GB BF16 VRAM (1×H100), output speed 36.9–222.4 token/s, ~$0.40 per million output tokens.
• Qwen3.8-27B (dense hybrid): total params 27B, active params 27B, ~56 GB BF16 (1×H100), output speed 46.8 token/s, ~$3.00 per million.
• Nemotron 3.5 Lightning (MoE + Mamba-2 hybrid): total params ~30B, active params ~3B, ~60 GB BF16 (1×H100), output speed 235.7–494.2 token/s, ~$0.22 per million.
• Mistral Small 4 (MoE, multimodal): total params 119B, active params 6B (8B incl. embeddings), ~121 GB FP8 (4×H100), output speed 147.3 token/s, ~$0.60 per million.

These numbers show that models with similar total parameter counts can have very different throughput and cost profiles depending on whether they are dense or MoE and on additional design choices. For instance, Nemotron Lightning’s activation of ~3B parameters per token is a major factor in its high throughput, alongside Mamba-2 layers and speculative decoding.

When to choose dense or MoE

Consider these factors when picking an architecture for deployment:

• Memory budget: memory footprint follows total parameters, so a 30B MoE and a 30B dense model need comparable VRAM. The practical difference is what those gigabytes buy you: dense models turn them into capability, MoE into throughput.

• Concurrency: MoE wins clearly for single-request throughput. Its throughput advantage usually persists as concurrency increases, but latency differences shrink. If you are latency-sensitive under high concurrency, benchmark both.

• Fine-tuning plans: dense models are easier to fine-tune because all parameters activate and gradients flow uniformly. Full fine-tuning on an MoE can unbalance the router (some experts become overused or inactive). LoRA/PEFT and freezing the router are common mitigations. NVIDIA NeMo provides a supervised fine-tuning recipe tailored to Lightning.

• Quantization: check the checkpoint’s native precision (for example, Mistral Small 4 ships natively in FP8, so 4-bit quantization yields less-than-4× memory reduction). Different modules quantize poorly across architectures: MoE routers are sensitive to small perturbations that can flip routing decisions, while hybrid-attention models can have other sensitive projection layers.

Core trade-off

Dense and MoE address the same trade-off from different angles: capability per parameter versus compute per token. Dense keeps everything active and simpler to serve and fine-tune. MoE buys throughput by paying memory up front and accepting greater serving complexity. The correct choice depends on workload: MoE is attractive for high-volume, well-specified agentic execution, while dense models may be preferable when single hard reasoning passes determine outcomes.

Nemotron 3.5 Lightning is fully open with weights, data, and recipes so organizations can adapt it and deploy it where they need; weights are available from Hugging Face and ModelScope and the model can be tried via build.nvidia.com or OpenRouter. For physical AI workloads, AgiBot GO-1 and Tencent Hy-Embodied-VLM-1.0 are other noted options in the ecosystem.