Growing model sizes and more complex reasoning tasks drive rising compute demand in AI. Hyperscalers and AI-native companies are increasingly developing custom accelerators—XPUs—to meet those needs. Scaling these accelerators requires high-bandwidth memory (HBM) to keep compute units fed, adequate package and silicon area for extra compute, efficient power delivery, and a resilient supply chain. Rack-scale architecture is also necessary to integrate XPUs into data center infrastructure.
NVIDIA NVLink Fusion is the connectivity technology and IP that lets hyperscalers and AI-native companies deploy custom XPUs and CPUs into the NVIDIA AI infrastructure platform. Using NVIDIA’s scale-up and scale-out technology stack, ecosystem, and MGX rack-scale architecture can reduce development and deployment complexity, improve performance, and accelerate time to market for semi-custom AI factories.
At the package level, NVHBM complements that unified architecture. NVHBM is a custom HBM base-die technology designed and validated with leading memory vendors to increase memory bandwidth, save area, and lower power consumption—improvements intended to help custom XPUs support larger models, read KV cache data faster, and improve training and large-scale inference.
Why bandwidth, die area and power matter
Training, inference, and agentic AI workloads increasingly depend on high-throughput access to model weights, KV cache, and activation data. As systems scale from individual accelerators to rack-level compute domains, a balancing act is required among compute logic, power delivery, thermal design, and high-bandwidth memory.
HBM places vital memory bandwidth close to the accelerator, but qualifying leading memory technology, package integration, and validation can become bottlenecks for custom accelerator programs. Through NVLink Fusion, customers can access NVHBM base dies that are validated with leading memory manufacturers, helping to reduce integration and qualification delays.
Key NVHBM benefits stated by the vendor:
- Bandwidth: up to 30% more memory bandwidth per stack compared with standard HBM4e.
- Area: more efficient interface connections can free up to 25% more main-die area for additional XPU capabilities.
- Power: up to 15% lower HBM power usage versus standard HBM4e.
The memory bandwidth bottleneck in modern AI accelerators
Accelerator performance depends on how consistently compute engines are supplied with data. Higher HBM speeds increase usable memory bandwidth within a given package budget, improving the ability to serve bandwidth-intensive phases of training and inference. NVHBM delivers up to 30% more memory bandwidth per stack compared with standard HBM4e, which for memory-bound or partially memory-bound workloads can translate into better accelerator utilization and higher throughput.
Faster memory transfers between HBM and compute cores can increase per-user token throughput during large-model inference by keeping compute units fed. While NVHBM raises memory bandwidth on each accelerator, NVLink Fusion connects accelerators across larger domains so workloads can make more efficient use of distributed compute and memory.
This scale-up domain is particularly important for advanced routing techniques such as expert parallelism (EP) or WideEP, where different experts live on different GPUs and require high-speed synchronization across a rack. NVIDIA NVLink, presented as the scale-up networking fabric for AI factories, moves activations and hidden states between experts and helps synchronize distributed caches; NVHBM reduces data starvation by maintaining local compute feed.
More package area, more flexibility
In custom AI silicon every square millimeter matters. Designers must allocate area among matrix engines, vector units, on-chip SRAM, cache hierarchy, control logic, memory interfaces, network-on-chip, and scale-up connectivity. A custom memory implementation can cut the design and package overhead tied to HBM access, freeing area for workload-specific capabilities.
NVHBM’s area savings come primarily from a redesigned physical memory interface (PHY). Standard HBM uses wider interface connections that increase package footprint. NVHBM uses a custom base die optimized for efficiency, with reduced I/O area achieved by moving the memory controller into the 3D HBM stack and integrating a custom PHY. Compared with the JEDEC HBM4e standard, this approach reduces PHY and support area by up to 67% and, by simplifying interposer routing, can provide up to 80% more usable silicon across the full layout.
The freed-up space allows the central AI compute die to expand into newly available area: the design claims up to a 30% increase in available main-die silicon for compute or other features, enabling XPU designers to add capabilities within a fixed package footprint.
Power savings for efficient scaling
Power is among the strictest constraints in modern AI infrastructure. HBM power contributes to an accelerator’s total power budget, package thermal design, rack power envelope, and data center cooling. NVHBM is said to enable 15% lower HBM power usage compared with standard HBM4e, creating additional power and thermal headroom for compute.
Those savings matter at multiple levels: at the XPU level they can improve performance per watt or permit more compute; at the rack level they ease pressure on power delivery and cooling; at data center scale modest reductions across thousands of accelerators compound to significant totals. The source states that, in a 1-gigawatt data center using 2,000 W XPUs, the power savings could enable up to 15,000 additional XPUs in compute headroom.
For large-model inference this is particularly relevant, since XPUs repeatedly read model weights and KV-cache while serving users at low latency and high throughput. Reducing the energy spent moving that data can support faster inference on large models, larger batch sizes, and more efficient use of deployed power.
Combining NVLink Fusion with NVHBM at rack scale
NVHBM improves XPU performance and efficiency at the chip level, while NVLink Fusion enables hyperscalers and AI natives to connect XPUs to the rest of the NVIDIA AI platform. By compounding up to 30% more memory bandwidth, up to 25% more die area, and up to 15% HBM power savings, these co-designed architectural improvements are presented as translating into up to a 30% overall end-to-end performance increase per XPU.
This connectivity is implemented via the NVLink Fusion chiplet, which bridges custom XPUs and the NVLink fabric to connect all XPUs in a rack into a single scale-up domain. NVLink is now in its sixth generation and is described as a proven, purpose-built scale-up networking fabric for AI factories. Upstream connections allow XPUs to link to CPUs via NVLink-C2C.
Adopters of NVLink Fusion can combine custom XPUs and CPUs with NVIDIA’s scale-up and scale-out stack and ecosystem to reduce development and deployment complexity, increase performance, and speed time to market for semi-custom AI factories. Standardizing on a single unified architecture also aims to simplify operations across the data center, enable flexible reprovisioning of capacity, and allow custom XPUs to integrate with GPUs for heterogeneous compute.
The next phase of custom AI silicon
NVLink Fusion provides a common scale-up foundation for GPUs, custom XPUs and CPUs, networking, and rack-level software. NVHBM complements that foundation with higher memory performance, increased compute density, HBM power efficiency, and supply resiliency for next-generation accelerators. Together, these technologies are presented as giving partners a more direct path from custom AI accelerator design to rack-scale deployment and production volume.
For further details the vendor points to additional materials on NVLink Fusion and industry adoption of NVHBM.
