Recommender systems (RecSys) are widespread in consumer internet services but are difficult to train and serve at scale. The rise of large language models (LLMs) has prompted an architectural shift: generative recommenders (GRs) treat recommendation as a sequence modeling task, predicting the next action or item from a user history sequence. This article explains that shift, the production challenges it creates, and how NVIDIA's recsys-examples and nv-embedding-cache projects address them.
Why traditional RecSys fails at scale
- Data type and volume: user histories mix categorical and continuous features and, at industry scale, can reach terabytes or petabytes per day. Such volumes rarely fit in a single GPU's high-bandwidth memory (HBM), causing training and inference bottlenecks.
- Sparsity and the long-tail: a small set of popular items tends to concentrate most interactions, leaving many niche items with little training signal.
- Cold start: new users or items lack interaction history, so embeddings must be inferred from limited features, which can degrade early recommendations.
- Strict latency: production RecSys often must retrieve and rank thousands of candidates within a few milliseconds under SLAs, unlike some LLM tasks that can tolerate autoregressive decoding latency.
Generative recommenders (GR)
GRs model the conditional probability of the next item given a user history: P(next_item | user_history) This reframing favors more homogeneous, transformer-like architectures that can exploit scaling laws and potentially unify retrieval and ranking into a single model. Two prevalent implementations are HSTU and Semantic IDs.
HSTU
Hierarchical Sequential Transduction Units (HSTU), introduced by Meta in 2024, reformulate RecSys as a generative objective. HSTU represents input as per-user sequences of interleaved items and actions ordered by timestamp, and it replaces explicit feature engineering with learned sequential representations via attention over user-item interactions. HSTU also alters standard Transformer attention by replacing softmax with SiLU-based weighting, adding relative attention bias, and applying elementwise gating before projection. These changes preserve stronger magnitude information across long sequences and enable more efficient kernel fusion and lower-latency inference.
Semantic IDs
Semantic IDs (SIDs), introduced by Google, reduce the effective vocabulary by hierarchically clustering item embeddings into a smaller token set. Several modern GR architectures (e.g., TIGER, PLUM, OneRec v1/v2) use Semantic IDs. Autoregressive decoding of Semantic IDs generates recommendations directly and yields a ranking through output logits; beam search can generate multiple Semantic IDs per forward pass, improving throughput and enabling selection of niche items within clusters.
recsys-examples repository
recsys-examples is a collection demonstrating best practices for training and deploying generative recommenders on NVIDIA GPUs using PyTorch. It contains optimized HSTU and Semantic ID implementations and three modular components: DynamicEmb (dynamic embeddings), a KV cache and storage manager for recommenders, and efficient CUDA ops for HSTU and Semantic ID beam search decoding.
Dynamic embedding (DynamicEmb)
Static embedding tables are impractical in production because new users and items constantly appear and the long-tail grows faster than a single GPU HBM. DynamicEmb implements a GPU-optimized scored hash table that maps arbitrary feature IDs to embedding rows on demand; rows are allocated only for seen IDs and the table spans HBM and pinned host memory. Based on HierarchicalKV algorithms, admission control and score-based eviction focus capacity on IDs that matter. DynamicEmb is provided as a TorchRec backend, with row-wise sharding via EmbeddingBagCollection and EmbeddingCollection. Fused CUDA kernels handle lookup and gradient reduction for SUM, MEAN, and sequence pooling, and prefetching keeps frequently accessed embeddings resident in HBM.
HSTU support
recsys-examples provides a production-style training and inference stack for HSTU. Item, user, action, and contextual embedding tables are managed through TorchRec, while DynamicEmb supplies dynamic capacity and caching. For dense layers, the HSTU backbone uses Megatron-Core, enabling data, tensor, sequence, and pipeline parallelism in a single training run. The modular design allows plug-and-play components.
During training, TorchRec/DynamicEmb and Megatron-Core coordinate sharding and parallelism across both embedding and dense modules. The training pipeline includes dynamic shuffling, overlaps embedding communication and prefetching with dense computation, and includes an HSTU layer with fused ops and FBGEMM attention kernels optimized for NVIDIA Ampere, Hopper, and Blackwell GPUs. These optimizations increased end-to-end Model FLOP Utilization (MFU) from 7.65% to 31.40% on two DGX H100 nodes — a substantial training efficiency gain.
For inference, recsys-examples supports low-latency requirements via PyTorch AOTInductor in the Torch C++ runtime and retains compatibility with NVIDIA Triton Inference Server. Frequently accessed embeddings are kept close to the GPU using nv-embedding-cache, and computation is reduced with a FlexKV-enabled KV cache that spans memory tiers. Measured Triton deployment results show that the PyTorch AOTI backend without KV cache yields a 1.14×–1.28× speedup over the Python backend, and with KV cache in an ideal all-GPU cache-hit scenario delivers a 2.20×–2.38× speedup.
Semantic ID–GR inference
Semantic ID GR serving differs from chat-style LLM inference: it typically has very long user context, short autoregressive decoding, and large beam widths over constrained item-token spaces. Practical workloads can contain thousands of history tokens, decode only 2–3 Semantic ID tokens, and use beam widths such as 128 or 256. General LLM serving systems (vLLM, SGLang, TensorRT LLM) are optimized for chat workloads and may not expose the GR-specific abstractions needed: shared request-level ContextKV, short per-beam BeamKV, beam-path tracking, dynamic beam width, and item-constrained generation. To address this, recsys-examples provides a GR-specialized inference framework example for Qwen-based Semantic ID models. The framework separates KV cache into ContextKV (long shared context), BeamKV (short decode history), and BeamPath (logical beam ancestry) to avoid treating every beam as a separate long sequence. The runtime includes GR-native continuous batching, CUDA graph replay, item-constrained topK, and a gr-decode_atten backend operating on the GR KV layout.
Semantic ID benchmark results
On a single NVIDIA H100 80GB GPU with Qwen3-1.7B, with context lengths of 1,000 and 5,000, beam width 256, and 3 output tokens, the GR-specialized path consistently outperformed SGLang beam-search in offline and online benchmarks. Selected measurements:
- Offline latency (ctx=1000, batch=4, beam=256, output=3): GR = 47.736 ms vs Baseline = 102.318 ms — ~2.14× faster
- Offline latency (ctx=5000, batch=4, beam=256, output=3): GR = 154.224 ms vs Baseline = 349.857 ms — ~2.27× faster
- Offline latency (ctx=5000, batch=8, beam=256, output=3): GR = 307.917 ms vs Baseline = 685.354 ms — ~2.23× faster
- Online serving throughput (ctx=5000, concurrency=4, beam=256, output=3): GR ≈ 19.7 req/s vs Baseline ≈ 10.7 req/s — ~1.85× higher
- Online median latency (ctx=5000, concurrency=4, beam=256, output=3): GR ≈ 198 ms vs Baseline ≈ 370 ms — ~46% lower
- Offline correctness: Top1 exact 1.000; TopK overlap mean 0.960 vs SGLang.
These results indicate the Semantic ID-GR inference path effectively complements HSTU and embedding components: HSTU and DynamicEmb address training scale and embedding-heavy inference, while the Semantic ID path targets autoregressive Semantic ID generation with long context and large beam searches.
nv-embedding-cache (NVE)
nv-embedding-cache is an SDK for accelerating large-scale embedding table lookups and operations in recommender inference. It supplies optimized kernels, cache primitives, and PyTorch-compatible bindings to enable low-latency access to embedding tables that exceed a single GPU's HBM.
NVE implements a hierarchical lookup flow: GPU cache in HBM (hot tier), CPU cache in DRAM (warm tier), and a remote parameter store (e.g., Redis or RocksDB) for cold storage. Hot-key promotion into the GPU cache is configurable, and a lockless invalidate-and-commit protocol lets lookups and cache updates run concurrently on the GPU without stalling lookups. Sharding across devices is managed via CUDA virtual memory so a single logical table may span multiple GPUs or nodes.
NVE offers NVEmbedding and NVEmbeddingBag as drop-in replacements for torch.nn.Embedding and torch.nn.EmbeddingBag, with cache-specific configuration flags. Because these layers behave like standard nn.Module components, they integrate into existing models with minimal graph changes. NVE also registers lookup operators against the LibTorch Stable ABI and supports AOTInductor in the C++ runtime. recsys-examples DynamicEmb tables are supported in NVE for an easy training-to-inference transition.
On DLRM v3 (the MLPerf generative recommender benchmark based on HSTU), recsys-examples together with NVE achieved 99,997 queries/sec inference throughput in an online server scenario.
Getting started
recsys-examples and nv-embedding-cache repositories include quick-start guides and documentation to implement and deploy these components. Used together, they enable industrial-scale training and low-latency recommender inference on NVIDIA GPUs.
Conclusion
Generative recommenders reframe recommendation as next-item prediction, improving architectural homogeneity and unification potential but raising production challenges: massive sparse embeddings, cold-starts, and strict latency SLAs. NVIDIA's recsys-examples and nv-embedding-cache provide DynamicEmb, HSTU-optimized kernels, GR-specialized inference, and hierarchical embedding caching that collectively demonstrate measurable improvements in training MFU, inference latency, and throughput across the presented benchmarks.
