Profiling Inclusion AI’s Ling-3.0-flash in batch-size-1 evaluation on NVIDIA DGX Spark showed decoding is memory-bound: reading model weights dominated runtime. Two operations were the primary contributors to execution time: the mixture-of-experts (MoE) computation and the output-layer projection (LM head) that produces logits across the entire vocabulary. This suggested two improvement directions: reduce weight traffic per decode step and increase the number of tokens produced per verification round.
This article examines three practical optimizations addressing those costs:
- Replacing FlashInfer’s MXFP4 MoE backend with Humming MXFP4 kernels to speed routed-expert computation.
- Applying online FP8 quantization to the LM head to halve its weight-read volume.
- Using multi-token prediction (MTP) speculative decoding via NEXTN to draft and verify multiple tokens per round.
Ling-3.0-flash is used as the implementation example; code and deployment instructions are provided in the Ling cookbook.
Model and baseline configuration
The tests used Ling-3.0-flash-fp4, a mixture-of-experts model with 124 billion parameters and 5.1 billion active parameters per token. The model has 42 layers: 35 Kimi Delta Attention (KDA) linear-attention layers and seven multi-head latent attention (MLA) layers. Each MoE layer routes tokens to eight of 512 routed experts plus one shared expert.
Ling-3.0-flash offers a native 256K-token context window, expandable to 1M tokens. Compared with the previous Ling flash model, it improves long-horizon stability, tool-calling accuracy, and harness compatibility. Developers can enable enhanced reasoning for agent tasks or disable it for latency-sensitive interactions.
The baseline followed the official cookbook W4A8 configuration: FlashInfer MXFP4 MoE kernels, NVIDIA CUTLASS dense GEMM, and the –enable-fp32-lm-head option. The LM head projects a hidden dimension of 2,560 to a vocabulary of 157,184 tokens.
Refer to the MXFP4 deployment notebook for standard MXFP4 deployment steps.
Optimization 1: Humming MXFP4 MoE backend
The first change swaps the flashinfer_mxfp4 MoE backend for humming, an independent kernel library available as the humming-kernels pip package. Unlike FlashInfer’s CUTLASS path, Humming uses NVIDIA CUDA Runtime Compilation (NVRTC) to generate and compile CUBINs at runtime for the actual M/N/K dimensions, data types, and GPU architecture; it caches the compiled binaries under ~/.humming/cache. This specialization better fits Ling-3.0-flash shapes (for example, its moe_intermediate_size of 768) and compiles kernels at finer granularity—individual kernels complete in seconds rather than requiring ninja to compile many targets, which can take tens of minutes.
Performance is reported as time per output token (TPOT) and decode throughput (tokens/second). On a synthetic workload (32K input tokens, 2K output tokens, batch size 1):
- Baseline: TPOT 27.71 ms, throughput 36.09 tok/s
-
- Humming MoE backend: TPOT 25.35 ms, throughput 39.45 tok/s (+9.3% throughput)
Optimization 2: Online FP8 LM head quantization
In the checkpoint the LM head is stored as BF16 and listed in modules_to_not_convert, so it is not quantized at load time. The vocabulary GEMM is 157,184 × 2,560 and the full weight block must be read for every generated token. Under a bandwidth bottleneck this weight block is an obvious compression target.
The optimization applies dynamic FP8 quantization only when constructing the LM head; the rest of the model continues to use the checkpoint’s FP8/MXFP4 configuration. Converting the LM head from BF16 to FP8 roughly halves the read volume for that layer. The setting is enabled with SGLANG_ENABLE_FP8_LM_HEAD=1.
Note that this option is mutually exclusive with –enable-fp32-lm-head, which outputs logits in FP32 and uses a slower vocabulary-GEMM path.
Cumulative effect (with Humming active):
-
- Online FP8 LM head: TPOT 22.76 ms, throughput 43.94 tok/s
- Relative to baseline: +21.7% throughput
- Relative to Humming only: +11.4% throughput
Optimization 3: MTP speculative decoding with NEXTN
The first two optimizations reduce the cost per decode step; speculative decoding reduces the number of steps. Ling-3.0-flash includes a NEXTN prediction layer (num_nextn_predict_layers: 1) that can serve as a draft model without further training or downloads. The configuration used three draft steps and a four-token verification window: the draft predicts three tokens sequentially and the target verifies four positions at once, accepting any correct prefix.
Benefit depends on accept length (average accepted tokens per round) and the verification-cost ratio (cost of one verification step relative to a normal decode step). The measured verification-cost ratio was about 2.26. At batch size 1 the main verification cost was not attention (KV reads were small relative to weight reads) but MoE expert activation: a normal decode activates eight experts for one token, whereas verification over four tokens can activate up to 32 experts.
Integration caveat: when online FP8 LM-head quantization and MTP are both enabled, the draft model must share the entire LM-head module with the target, not just the weight tensor. The original Ling NEXTN implementation did not expose that interface. Without it the system silently falls back: the draft gets FP8 weights but interprets them as BF16, accept length drops to 1.00, and throughput becomes worse than the unoptimized baseline. Sharing the full LM-head module restores normal token acceptance. The Ling cookbook documents how to enable FP8 LM-head quantization and MTP together.
Cumulative effect (Humming + FP8 active):
-
- MTP (NEXTN, steps=3 / topk=1 / draft_tokens=4): TPOT 9.72 ms, throughput 102.88 tok/s
- Relative to baseline: +185.1% throughput
- Synthetic accept length reached 3.90 tokens per round (note synthetic conditions overestimate gains compared with real tasks)
End-to-end results on GSM8K
The component-level results used a synthetic 32K-input/2K-output workload. Because synthetic workloads tend to be repetitive and use simpler sampling, they overestimate speculative decoding benefits. Therefore the evaluation also included an end-to-end test across all 1,319 GSM8K questions.
Setup: five-shot prompting (demonstrations from training set), full test set evaluation, temperature 0.6, top_p 0.95, top_k 29, max_tokens 32,768, chain-of-thought enabled. Aside from the three optimizations, all variables were aligned (sampling parameters, parser, five-shot protocol, random seed, memory allocation). Time to first token (TTFT) was measured.
Results across GSM8K (batch size 1):
- TPOT: baseline 26.45 ms → optimized 18.56 ms (−29.8%)
- Decode throughput: baseline 37.81 tok/s → optimized 53.89 tok/s (+42.5%)
- GSM8K accuracy: baseline 96.89% → optimized 96.82% (−0.08 percentage points; no statistically detectable accuracy loss)
- MTP accept length on GSM8K: 2.68
- Mean latency: baseline 15.21 s → optimized 11.49 s (−24.5%)
- Latency P50 / P95: baseline 8.36 s / 50.20 s → optimized 6.11 s / 38.60 s (−27% / −23%)
- Mean TTFT: baseline 0.207 s → optimized 0.214 s
- Request errors / truncations: 0 / 0 in both runs
Notes on differences between synthetic and real workloads: synthetic outputs are more repetitive (leading to higher accept lengths: 3.90 synthetic vs 2.68 GSM8K), sampling differed (synthetic used greedy decoding whereas GSM8K used temperature/top_p/top_k), and output-length distributions differ (synthetic fixed at 2K tokens while GSM8K median ~310 tokens). These factors mean synthetic throughput gains (e.g., 185.1%) are not guaranteed on other workloads.
Conclusions and deployment guidance
Across the full GSM8K evaluation, the three optimizations together increased decode throughput by 42.5%, reduced mean end-to-end latency by 24.5%, and showed no measurable accuracy change. Mechanistically they do not overlap: Humming and FP8 LM head reduce per-step cost, while MTP reduces step count, so combining them is sensible.
However, stacking optimizations requires correct integration. Each optimization worked independently, but enabling all three required the draft model to share the complete LM-head module; otherwise the system can silently degrade to a configuration slower than the unoptimized baseline. During deployment, explicitly validate that each optimization is active at startup rather than inferring their state from later performance numbers. The Ling cookbook provides instructions for enabling Humming, online FP8 LM-head quantization, and MTP together.
