The transformers library is the reference modeling library in machine learning, supporting over 450 architectures with consistent APIs. It is designed so that model implementations are self-contained and easy to read, which makes it straightforward to learn an architecture from the transformers code and port it to other runtimes such as vLLM.
vLLM previously integrated transformers as a modeling backend so authors didn’t need to port models: transformers provides the model definition and vLLM supplies highly optimized inference techniques like continuous batching and custom attention kernels. The recent improvements further tighten that integration.
What the update delivers
The transformers modeling backend for vLLM now matches or exceeds the throughput of vLLM’s hand-written native implementations for several tested models. The team compared three distinct Qwen3 variants:
- Qwen3-4B dense on a single GPU
- Qwen3-32B dense with tensor-parallel across 2 GPUs
- Qwen3-235B-A22B-FP8 Mixture-of-Experts (MoE) with data- and expert-parallelism on an 8×H100 node
The outcome: in each case the transformers backend met or beat the native vLLM throughput. As a result, running most Hugging Face models through the transformers backend requires only a single flag:
- --model-impl transformers
This composes with existing parallelism flags so your serving setup doesn’t need to change. Examples from the announcement:
- Qwen3-4B on one GPU: vllm serve Qwen/Qwen3-4B --model-impl transformers
- Qwen3-32B, tensor-parallel on 2 GPUs: vllm serve Qwen/Qwen3-32B --model-impl transformers --tensor-parallel-size 2
- Qwen3-235B MoE, data + expert parallel across 8 GPUs: vllm serve Qwen/Qwen3-235B-A22B-FP8 --model-impl transformers --data-parallel-size 8 --enable-expert-parallel
Note: models using linear attention are not currently supported but are planned to be added. Custom models whose code lives in a Hub repo are unlikely to work unless they were written compliantly.
How the comparison was done
Each model was benchmarked under three otherwise-identical conditions differing only by the code path:
- native — --model-impl vllm (the baseline)
- after — --model-impl transformers (with the PR)
- before — --model-impl transformers (without the PR)
A reproducible benchmark runner script is provided so measurements can be repeated.
What changed technically?
Previously the transformers backend emphasized attention as the primary bottleneck and plugged in vLLM’s attention implementation at runtime. However, peak inference performance depends on many other factors—multi-GPU parallelization, fused kernels, compilation, and more—which had required custom vLLM implementations to fully exploit.
The new approach dynamically applies inference-specific layer fusions at runtime for compatible architectures, allowing transformers implementations to reach the same speeds as custom vLLM code without manual porting.
How it works
The transformers modeling backend now uses torch.fx to perform static analysis on the model graph to find known, optimizable patterns. After pattern identification, it uses the Python AST (abstract syntax tree) tools to manipulate and rewrite parts of the source code in place at runtime.
These rewrites produce fused operations that map many-to-one onto highly optimized vLLM kernels — for example, kernels used for Expert Parallelization (EP) in MoE models. Major fused blocks include vLLM’s MergedColumnParallelLinear and QKVParallelLinear, which enable inferring tensor-parallel (TP) plans; pipeline-parallel (PP) plans can also be inferred when decoder block lists are identifiable.
The transformed models remain fully torch-compilable and can go through torch.compile and CUDA Graphs, the same as a native vLLM model implementation.
Additional benefits
Unlike dedicated vLLM model implementations, transformers model implementations are usable for training as well. That means the same model code can be used for training, evaluation, RL rollouts—and now, for inference at native vLLM speed for compatible models.
The team is preparing a detailed blog post that will dive deeper into the optimized inference methods and the model manipulation techniques.
Resources
- Transformers model definition
- Transformers modeling backend in vLLM
- Large scale serving
- Torch FX
- Abstract syntax tree
Note: models with linear attention are not yet supported; custom Hub-based model code will likely fail unless written to the compatibility guidelines.
