Baseten, an AI infrastructure company focused on inference engineering, has raised $13 billion and joined a new wave of decacorn AI infrastructure firms. Inference engineering—the discipline of optimizing trained models for speed, reliability, and cost in production—has emerged as critical over the past three years.
Recent advances show that careful quantization and error cancellation can boost throughput 20% while preserving model quality, and optimized serving can make models up to 10× faster.
Baseten exemplifies this shift by offering cache-aware routing, disaggregated compute, and day-zero support for new open models, making inference optimization a core competitive advantage.
What happened
Baseten has raised $13 billion and is now one of a new cohort of AI infrastructure "decacorns" (billion-dollar companies). The company, co-founded by Philip Kiely and staffed by engineers like Ali Taha, is becoming central to the shift toward inference engineering—the discipline of turning trained model weights into fast, reliable, and affordable products at scale.
Why it matters
Inference engineering has emerged as a critical discipline over the past three years. Recent breakthroughs show that quantizing more of a model (reducing precision) can increase throughput by 20% while preserving benchmark quality, because errors in different layers can cancel each other out. This means AI companies, including Nvidia, were previously sacrificing model intelligence and speed unnecessarily. Baseten's focus on cache-aware routing, disaggregated prefill and decode, speculative decoding, and other optimizations allows models to run up to 10× faster—making frontier open models viable alternatives to closed systems for many use cases.
What to watch
Baseten is now demonstrating production-ready support for cutting-edge open models on day zero. The company recently grafted Kimi's vision encoder onto GLM-5.2 without changing the underlying language model, showing how inference engineering can fuse components from different architectures. The field is converging with systems design: hardware, kernels, and infrastructure choices are becoming as important as model architecture itself.
Baseten, the AI infrastructure company behind the book "Inference Engineering," has raised a $13 billion funding round, joining a new cohort of AI infrastructure decacorns alongside Nvidia, Intel, and others. The company is led by Philip Kiely, an author and speaker on inference engineering, and employs Ali Taha, an engineer known for viral technical breakdowns of cutting-edge models like Kimi K3.
Inference engineering—the discipline of turning trained model weights into fast, reliable, and affordable products at scale—barely existed as a category three years ago but has become one of the most critical disciplines in AI. Unlike model training, which asks "can we build weights that work?", inference engineering asks "how do we deploy those weights efficiently in production?" The field tackles a distinct optimization problem that creates new research challenges, infrastructure requirements, and specialized engineering roles.
Baseten's technical approach to inference centers on several key optimizations. When a user sends a long query (such as a 200,000-token request), Baseten employs cache-aware routing to check whether part of the query has been seen before, allowing the system to reuse previously computed KV cache and skip recomputation. For models handling very long inputs, the company has disaggregated prefill and decode—using separate GPU pools to process input (create the KV cache and generate the first token) and then iteratively generate subsequent tokens. Speculative decoding, which uses a smaller model tuned to the workload (e.g., coding) to draft tokens for acceptance by a larger model, further accelerates generation. For high-volume users processing millions of tokens per hour, Baseten offers dedicated deployments where customers pay per hour rather than per token, which is often substantially cheaper and also allows customization of inference optimizations for specific traffic patterns.
Recent breakthroughs in quantization—reducing model precision to lower compute costs—have shown unexpected gains. In one GLM-5.2 experiment, quantizing more of the model actually increased throughput by 20% while preserving downstream benchmark quality, because errors introduced by quantization in different layers could cancel each other out. This discovery contradicts conventional wisdom and suggests that companies including Nvidia were previously unnecessarily sacrificing model intelligence and speed. Baseten's approach also includes retrofitting inefficient model layers with components from other architectures; the company grafted Kimi's vision encoder onto GLM-5.2 without changing the underlying language model, demonstrating the flexibility of inference-time engineering.
The article indicates that inference optimizations can still deliver gains of 20%, 100%, or even 200%, and that optimized serving can make a model up to 10× faster than baseline. These gains reflect both the immaturity of the field and the enormous economic value of continued advances. Baseten's ability to support new open models on day zero—launching production-ready APIs as soon as a model is released—is emerging as a competitive differentiator, making open-source models viable alternatives to proprietary systems for many use cases. The field is converging with systems design, where hardware, kernels, cache management, and infrastructure choices are becoming as important as model architecture itself.
Inference engineering has crystallized as its own discipline only within the past three years, yet it now sits at the center of AI infrastructure competition. The field addresses a fundamentally different question than model training: given weights that work on paper, how do you deploy them fast, reliably, and affordably? Baseten's $13 billion raise reflects investor recognition that this problem is now as economically important as training itself.
The breakthrough discoveries cited in the article reveal why inference engineering matters. Quantization—reducing model precision—was long thought to destroy quality; yet Philip Kiely and Ali Taha's work shows that errors introduced by quantizing different layers can interfere constructively, yielding 20% throughput gains while matching benchmark scores. This insight means that companies like Nvidia (mentioned in the article as having previously sacrificed speed or intelligence) were leaving massive optimization opportunities on the table. The convergence of these findings with Baseten's ability to graft Kimi's vision encoder onto GLM-5.2 without retraining demonstrates that inference engineering is becoming a systems discipline: optimizing hardware, kernels, cache reuse, and routing simultaneously.
The article suggests that inference optimization can still deliver gains of 20%, 100%, or even 200%—a range that underscores how immature the field remains and how valuable continued advances are. Baseten's ability to support new open models on day zero (as shown by their GLM-5.2 work) indicates that inference engineering is becoming a service moat: whoever masters these optimizations fastest can make open models competitive with proprietary ones.
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · takes 30 seconds · unsubscribe anytime
Ask AI anything about this article. Q&As are published on this page for other readers too.
The AI news that matters, in one minute each morning.
Sign up free