AI system performance is shaped by three interdependent objectives: accuracy (how well the model reasons and the quality of outputs), throughput (tokens/second served by a datacenter) and interactivity (felt responsiveness, dominated by latency). In practice these objectives conflict: high accuracy is less useful if responses are slow, and raw throughput matters little if every user sees lag.
This article focuses on throughput and interactivity and explains how model-design choices influence both without gratuitously sacrificing accuracy. If accuracy is fixed, throughput and interactivity form a two-dimensional Pareto frontier; the engineering goal is to push that frontier outward so both metrics improve.
The discussion centers on large language models (LLMs) and treats two stakeholder perspectives: the system deployer, who cares about fleet-level tokens/sec, and the user, who values low first-token and inter-token latency. A single response goes: prompt → first-token latency → first token → inter-token latency → next token… Models that are designed to align with hardware not only run faster but scale better and cost less.
Deployment regimes
Deployments vary along two axes: context length (short → long) and service goal (throughput-oriented → latency-oriented). Different quadrants require different optimizations:
- Long-context, throughput-oriented: time is dominated by attention; parallelism must focus on shortening attention-related work.
- Short-context, throughput-oriented: attention and FFN time are more balanced; expert parallelism (MoE) can be effective at scale.
- Latency-oriented: added model parallelism can shorten critical paths but introduces communication and fixed overheads.
Amdahl’s law applies: optimize what dominates runtime. If attention accounts for 77% of runtime, tuning FFNs brings only marginal gains.
Why linear-layer dimensions matter (H, H′, L)
Transformer design choices—hidden dimension H, FFN intermediate dimension H′ and number of layers L—determine how compute and memory are distributed, how well a model maps onto parallelism, and how effectively it saturates GPUs.
Arithmetic intensity and the roofline model
Achievable performance is bounded by the roofline model. A workload’s arithmetic intensity (operations per byte moved) determines if it is memory-bound (low intensity) or compute-bound (high intensity). For maximum throughput you want the workload in the compute-bound region so the device’s peak FLOPS are utilized. Latency-sensitive decoding runs at low concurrency, is often memory-bound, and benefits from reducing memory-access time.
A GEMM C = A B with shapes M×K (A) and K×N (B) requires about 2 M N K FLOPs; read/write bytes scale with M K, N K and M N times the per-element byte size. Tokens, H and H′ map into GEMM M, N, K for each transformer linear layer (Q/K/V input, attention output, FFN up/down projections). If H or H′ is small, the GPU spends relatively more time moving data than performing math, even when the token (M) dimension is large.
Concrete example: FFN-2 with H′=512 and H=8192 on GB300 (NVFP4 inputs, FP8 outputs) remains memory-bound at large token counts because the write cost and small reduction dimension dominate.
Practical consequence: model dimension choices matter as much as batch size for saturating compute.
Guideline 1:
- For fixed-parameter models, prefer near-square weight matrices and avoid making the projection or reduction dimension too small.
Tile quantization and GPU execution of GEMMs
GPUs split GEMM outputs into tiles computed by streaming multiprocessors (SMs). SMs can cooperate (clusterMMA) and groups of SMs can act as a cooperative grid array (CGA). Cooperation increases data reuse but also enlarges the effective tile, so dimensions must be multiples of a larger tile size to stay aligned. If a dimension is not a multiple of the effective tile, partially filled edge tiles still run full compute, wasting cycles and reducing throughput.
To avoid tile quantization waste, choose model dimensions that are multiples of tile sizes and aligned to cache-line widths. A multiple of 128 is a safe floor; 256 or 512 align better with clusterMMA and CGA tiles and capture higher throughput.
Guideline 2:
- Make model dimensions multiples of 128 at minimum; prefer 256 or 512 to match larger cooperative tiles.
Wider models often better for hardware
With the same parameter budget, wider models typically deliver higher arithmetic intensity and lower latency than deeper models due to greater weight reuse and a shorter sequential critical path. That makes wider architectures advantageous for both throughput- and latency-oriented goals. Depth still contributes to representational power, so prefer width only while accuracy remains acceptable.
Guideline 3:
- When choices exist, prefer fewer, larger operations over many smaller ones—wider transformer models are generally more hardware-friendly than deeper ones.
Quantization as a performance lever
Quantization improves both compute-bound and memory-bound workloads by increasing math throughput and decreasing memory traffic. Blackwell-class systems support NVFP4 alongside FP8 and FP16/BF16. NVFP4 applies fine-grained scaling (E4M3) to 16-value micro-blocks plus a per-tensor FP32 scale; this hierarchical scaling reduces quantization error while keeping 4-bit execution speeds.
On DeepSeek-R1 benchmarks, NVFP4 accuracy closely matches FP8 baselines—within about one point on most measures and matching or exceeding FP8 on SciCode, Math-500 and AIME 2024. NVIDIA provides end-to-end tooling—TensorRT Model Optimizer and LLM Compressor—to enable PTQ, QAT and advanced calibration for NVFP4 with minimal accuracy loss. NVIDIA Research has also studied pretraining with NVFP4 (Pretraining Large Language Models with NVFP4, 2025).
Guideline 4:
- When adding expensive operations, consider whether they can be quantized at deployment; design layers to benefit from low-precision execution.
Expert parallelism for MoE models
Throughput-oriented serving aims to maximize users served per GPU. For Mixture-of-Experts (MoE) models, expert parallelism (EP) is an effective strategy: run attention with data parallelism and distribute FFN experts across GPUs. DP avoids the AllReduce overhead of tensor parallelism (TP), which grows with concurrency and hurts throughput. DP naturally increases GEMM-M with more GPUs, improving utilization in sparse MoE FFNs.
Assuming uniform routing: GEMM-M = (global concurrency × top-k) / #experts
Where global concurrency = concurrent tokens per GPU × number of GPUs; top-k is experts activated per token (DeepSeek-R1 uses 8); #experts is total experts (DeepSeek-R1 uses 256). Widening EP helps even when per-expert batches are small, by increasing aggregate memory bandwidth for weight loads and reducing each GPU’s memory footprint.
Challenges are all-to-all communication and load imbalance; TensorRT-LLM Wide-EP provides high-performance all-to-all kernels and an adaptive load balancer to address these.
Guideline 5:
- Large, sparse MoE models can achieve high throughput when deployed with wide expert parallelism.
Pipeline parallelism and disaggregation
Pipeline parallelism becomes useful when prefill and decode are disaggregated. Chunked Pipeline Parallelism (CPP) splits both model layers across GPUs and input context into chunks that flow through the pipeline—this helps maintain high throughput while reducing first-token latency (FTL) on long contexts. CPP requires balanced pipeline stages; designing models with repeatable layer patterns helps achieve that.
Guideline 6:
- Design models with regular, repeatable layer patterns that partition into balanced pipeline stages to benefit from CPP.
Hybrid parallel strategies for latency targets
Latency-oriented deployments require less work per GPU or more GPUs. As concurrency shrinks, FFN latency can become memory-bound because GEMM-N and GEMM-K remain large. In low-concurrency regimes, attention and FFNs should be parallelized independently: FFNs via TP, EP or TP×EP; attention via TP and KV parallelism. TP is limited by the number of KV heads; beyond that the KV cache must be replicated.
Helix Parallelism (NVIDIA Research, 2025) shards the KV cache across the sequence dimension during attention, then reuses GPUs for TP×EP in FFNs; Helix is supported in TensorRT-LLM. Though hybrid schemes add communication, Blackwell NVL72’s high-bandwidth NVLink absorbs much of the cost and the rest can often be overlapped with computation.
Guideline 7:
- Decouple attention and FFN parallelization in model and deployment design so adding GPUs can strategically reduce the dominant bottlenecks and improve interactivity.
Next steps and checklist
When designing a new model, use these guidelines as a checklist:
- Keep dimensions near-square and aligned to 128 (ideally 256).
- Favor width over depth while preserving accuracy.
- Design for low-precision (NVFP4) execution.
- Use regular, repeatable layer patterns that map cleanly to pipeline parallelism.
Small design choices like these raise GPU utilization, yielding faster inference, higher throughput and better interactivity on the same hardware. For implementation, explore NVIDIA TensorRT Model Optimizer for NVFP4 quantization and TensorRT-LLM for expert, pipeline and Helix parallelism.
