The transformers library's modeling backend for vLLM can now match the inference speed of custom vLLM implementations through dynamic layer fusions applied at runtime. This means model authors can deploy transformers models directly in vLLM without writing custom optimization code, cutting development time and letting new models reach production-grade performance immediately.
Summaries like this, in your inbox every morning.
Sign up free →What happened
The transformers modeling backend for vLLM now achieves inference speed comparable to vLLM's custom hand-written implementations across multiple Qwen3 model sizes—from 4B dense models to 235B Mixture-of-Experts models—by using torch.fx and abstract syntax tree manipulation to apply layer fusions at runtime.
Why it matters
Model authors no longer need to write separate optimized code for vLLM; they can use their transformers implementations directly and automatically get native vLLM inference performance. This reduces duplicate engineering work and makes it easier for new models to reach production speed without custom porting.
What to watch
Models using linear attention are not currently supported but will be soon. The full optimization process uses torch.fx for static analysis to identify fusible patterns, then rewrites operations to map to vLLM's ultra-optimized kernels (MergedColumnParallelLinear, QKVParallelLinear, and expert-parallel kernels).
The transformers library has established itself as the reference modeling standard for machine learning, supporting 450+ architectures through consistent APIs. Previously, achieving ultra-fast vLLM inference required model authors to write custom vLLM implementations in parallel with their transformers code—a duplicative and time-consuming process. The transformers backend for vLLM, integrated last year, allowed transformers models to run inside vLLM without porting, but performance still lagged because the focus was primarily on optimizing attention operations.
The new iteration solves this gap by dynamically applying inference-specific layer fusions at runtime, using torch.fx and abstract syntax tree manipulation to identify and rewrite fusible patterns. Testing across three different Qwen3 models—a 4B dense model, a 32B dense model with tensor parallelism, and a 235B Mixture-of-Experts model with data and expert parallelism—showed that the transformers backend now meets or exceeds the throughput of hand-written native vLLM implementations. This means a model author needs to integrate their architecture only once (in transformers) and can immediately leverage vLLM's full inference optimization suite without writing a single line of custom code.
The transformation is significant for the ML ecosystem: it lowers the barrier for new models to reach production-grade serving speed, reduces engineering duplication, and lets model code be reused across training and inference workflows. The one current limitation is that models using linear attention are not yet supported, though the article indicates this gap will be closed soon.
No comments yet. Be the first to share your thoughts!
Log in to join the discussion
Get curated AI news from 200+ sources delivered daily to your inbox. Free to use.
Get Started FreeFree · takes 30 seconds · unsubscribe anytime
1 minute a day. The AI essentials.
200+ sources · Email / LINE / Slack
