vLLM transformers backend now matches hand-optimized speeds

The transformers library has established itself as the reference modeling standard for machine learning, supporting 450+ architectures through consistent APIs. Previously, achieving ultra-fast vLLM inference required model authors to write custom vLLM implementations in parallel with their transformers code—a duplicative and time-consuming process. The transformers backend for vLLM, integrated last year, allowed transformers models to run inside vLLM without porting, but performance still lagged because the focus was primarily on optimizing attention operations.

The new iteration solves this gap by dynamically applying inference-specific layer fusions at runtime, using torch.fx and abstract syntax tree manipulation to identify and rewrite fusible patterns. Testing across three different Qwen3 models—a 4B dense model, a 32B dense model with tensor parallelism, and a 235B Mixture-of-Experts model with data and expert parallelism—showed that the transformers backend now meets or exceeds the throughput of hand-written native vLLM implementations. This means a model author needs to integrate their architecture only once (in transformers) and can immediately leverage vLLM's full inference optimization suite without writing a single line of custom code.

The transformation is significant for the ML ecosystem: it lowers the barrier for new models to reach production-grade serving speed, reduces engineering duplication, and lets model code be reused across training and inference workflows. The one current limitation is that models using linear attention are not yet supported, though the article indicates this gap will be closed soon.