When to Use Encode–Prefill–Decode Disaggregation for Faster Multimodal Inference

Encode–Prefill–Decode (EPD) disaggregation is an inference optimization that separates the vision encoder stage from the LLM prefill and decode stages. It is most beneficial when requests are image‑heavy, outputs are short‑to‑medium length, or the LLM is smaller, MoE, or quantized. Using NVIDIA Dynamo, the authors report up to 5× faster time to first token (TTFT) and up to 7× faster end‑to‑end response times in favorable scenarios.

Dynamo is an open‑source inference framework for serving AI models in distributed environments. It implements EPD by isolating encoder workers (which produce vision embeddings) from PD workers (which consume embeddings and run the LLM), allowing each stage to batch, schedule, and scale independently.

Aggregated serving versus EPD

In aggregated serving, a single worker handles media preprocessing, ViT forward and projection, LLM prefill, and decode within one scheduling domain. This is simple and effective when media processing is a small portion of total work. When requests include many images or video, vision encoding can take hundreds of milliseconds and become a bottleneck. Because enkóder and LLM share the same GPU in aggregated setups, media‑heavy requests can delay their own prefill and contend with other prefill/decode work. Under mixed traffic, text‑only requests may also be forced to wait behind multimodal work they do not need.

EPD in Dynamo separates encoder and PD roles without fixing their physical placement. Encoder workers produce embeddings and send them to PD workers via NVIDIA Inference Transfer Library (NIXL). This separation enables independent batching, scheduling, and scaling of the two stages.

Three encoder placement topologies

• Aggregated: each GPU runs one aggregated worker responsible for all phases of the request lifecycle.
• Colocated encoder: encoder worker(s) run alongside a PD worker on the same GPU. This keeps GPU compute shared but separates request queues and batching. On homogeneous clusters, colocated encoders usually fit better because the vision encoder is lightweight and reserving an entire GPU for it would underutilize resources.
• Disaggregated (heterogeneous) encoder: encoder workers run on a lower‑cost GPU tier (e.g., RTX 6000D) while PD workers run on higher‑end GPUs (e.g., GB200). Embeddings are transferred to the PD tier through NIXL. In the reported tests two RTX 6000D GPUs hosted encoder workers and four GB200 GPUs hosted PD workers. The analysis found homogeneous disaggregated deployment generally underperforms colocated.

Hardware availability and workload characteristics determine whether and where the encoder should run.

Which factors determine EPD benefits?

EPD gains depend on how work divides among ViT forward, LLM prefill, and decode. It is most useful when vision encoding is a significant part of processing time or limits throughput. Key factors:

• Input media load: more images or higher resolution increase visual tokens and encoder work — EPD lets you scale encoders to avoid encoding bottlenecks.
• Output sequence length (OSL): longer outputs shift latency toward decode. TTFT gains from EPD generally hold, but end‑to‑end (E2E) gains shrink as OSL grows.
• Model size / precision: ViT compute is largely fixed; LLM compute falls with smaller models and lower precision. Smaller, MoE, and quantized models therefore benefit more from EPD; very large dense models see less gain.
• Mixed traffic: when text‑only and multimodal requests mix, aggregated prefill batches can force text requests to wait for ViT. EPD isolates encoder work so text requests can proceed without waiting.

EPD reduces TTFT and increases same‑SLO goodput only when encoder work is large enough to offset coordination and embedding transfer overhead.

Test environment

Most benchmarks used Qwen3.5 122B A10B NVFP4 except for precision ablation tests. The setup used four GB200 GPUs (with additional RTX 6000D GPUs in disaggregated experiments). Configurations:

• Aggregated: one TP1 aggregated worker per GB200.
• Colocated EPD: two encoder workers plus one PD worker per GB200.
• Disaggregated EPD: RTX nodes as encoder tier, GB200 as PD tier.

NIXL over UCX RC/TCP Ethernet was used for embedding transfer, measured peak 20 Gbps. Dynamo frontend parallel media decoding was enabled. The goodput SLO was inter‑token latency (ITL) under 100 ms.

Image‑heavy workload example

For requests with 10 images per request (capped 256 tokens per image) and OSL 1024:

• Colocated encoder TTFT fell by 58% compared to aggregated serving.
• Heterogeneous (disaggregated) encoder TTFT fell by 50%.

End‑to‑end improvements were modest because OSL 1024 means decode time dominates and is not reduced by encoder disaggregation. The heterogeneous tier, however, served 70% more traffic at the same latency SLO because encoder capacity was added without expanding GB200 capacity.

How image load and OSL impact gains

Across experiments with 5–50 images (fixed OSL), aggregated latency rose steeply with visual tokens while both EPD topologies remained nearly flat. Varying OSL from 128 to 2048 with five images held constant showed TTFT largely unchanged as OSL increased, but decode increasingly dominated end‑to‑end latency. Consequently, heterogeneous EPD end‑to‑end gain over aggregated serving narrowed from 20.3% to 5.2%; colocated EPD shifted from an 11.8% gain to a 2.5% regression under high OSL due to GPU contention with PD work.

Encoder disaggregation delivers the most value at high input media load across most scenarios; with light media load and long outputs, disaggregation can hurt end‑to‑end performance.

Model size and precision effects

Model‑size ablation on Qwen3.5 4B, 9B, and 27B showed the ViT parameter share decreased from 7.2% (4B) to 4.7% (9B) and 1.7% (27B). Colocated EPD delivered same‑SLO goodput of 2.62×, 1.50×, and 0.65× relative to aggregated serving for those configurations respectively — indicating EPD gain declines with model size and can fall below break‑even for large dense models.

Precision ablation: when both ViT and LLM used BF16, colocated EPD achieved 1.78× goodput versus aggregated. Quantizing only the active LLM weights to NVFP4 raised that ratio to 2.64×. NVFP4 accelerates LLM prefill and decode while ViT remains in BF16, increasing the relative share of encoder work and therefore EPD’s benefit.

EPD for mixed modality traffic

In mixed production traffic, aggregated serving can force text requests to wait for the encoder when a batch includes multimodal items. EPD runs vision encoding on separate encoder workers so text‑only requests can begin prefill immediately.

Measured on sustained 50:50 mixed traffic at OSL 128 (aggregated vs colocated EPD):

• Mean TTFT for text requests: 92.3 ms → 53.3 ms (42.2% reduction).
• Mean TTFT for image requests: 289.9 ms → 200.6 ms (30.8% reduction).

Text requests benefit more because they avoid the encoder head‑of‑line blocking.

How to choose the best encoder topology

Select encoder placement based on where request time is spent:

• Use EPD when vision encoding is a large portion of request processing (many images, short/medium outputs, smaller or quantized models, mixed traffic).
• Keep aggregated serving when media processing is light or very long decode sequences dominate latency.
• On homogeneous clusters, colocated encoders typically separate scheduling and batching without dedicating a GPU to encoders. On heterogeneous hardware, a separate encoder tier on lower‑cost GPUs can be preferable.

Note that vLLM and SGLang have roadmaps including further EPD stack development.

Getting started and complementary levers

To reproduce the experiments, follow the ai-dynamo/dynamo GitHub guide. As the authors point out, 43% of TTFT was spent before ViT started, so EPD does not affect those earlier pipeline stages. Dynamo also provides other levers to improve different stages:

• Parallel media decoding (--frontend‑decoding): moves download and decode to the frontend. In an encoder‑only benchmark (30×256 on one GB200 with two encode workers) it cut mean encoder request latency from 281.3 ms to 207.0 ms (‑26%) and P99 from 752.1 ms to 581.7 ms (‑23%) at matched throughput.
• Embedding cache: stores computed embeddings in CPU DRAM so repeated media is not re‑encoded.
• Multimodal KV routing: hashes media alongside text so requests sharing media land on workers already holding relevant KV blocks.

Summary

EPD disaggregation can materially reduce TTFT and raise same‑SLO goodput for multimodal inference when vision encoding is a sizable portion of work — notably for image‑heavy requests, short‑to‑medium outputs, and smaller or quantized models. Benchmarks with Dynamo showed large TTFT drops (50–58% for a 10‑image OSL 1024 case), a 42.2% TTFT reduction for text requests in mixed traffic, and up to 70% higher goodput for a heterogeneous encoder tier under some conditions. However, benefits diminish as decode time dominates (long OSL) or with very large dense LLMs; embedding transfer and coordination overheads must also be considered. Complementary Dynamo features (parallel decoding, embedding cache, KV routing) can further reduce pipeline latency.

Acknowledgments

Thanks to Roger Wang (Inferact) for feedback and to the NVIDIA team — Alexandre Milesi, Ayush Agarwal, Guan Luo, Indrajit Bhosale, J Wyman, Kris Hung, Krishnan Prashanth, Qi Wang, and Zhongdao Ren — for core work on multimodality support in Dynamo.