What happened
Baseten has raised $13 billion and is now one of a new cohort of AI infrastructure "decacorns" (billion-dollar companies). The company, co-founded by Philip Kiely and staffed by engineers like Ali Taha, is becoming central to the shift toward inference engineering—the discipline of turning trained model weights into fast, reliable, and affordable products at scale.
Why it matters
Inference engineering has emerged as a critical discipline over the past three years. Recent breakthroughs show that quantizing more of a model (reducing precision) can increase throughput by 20% while preserving benchmark quality, because errors in different layers can cancel each other out. This means AI companies, including Nvidia, were previously sacrificing model intelligence and speed unnecessarily. Baseten's focus on cache-aware routing, disaggregated prefill and decode, speculative decoding, and other optimizations allows models to run up to 10× faster—making frontier open models viable alternatives to closed systems for many use cases.
What to watch
Baseten is now demonstrating production-ready support for cutting-edge open models on day zero. The company recently grafted Kimi's vision encoder onto GLM-5.2 without changing the underlying language model, showing how inference engineering can fuse components from different architectures. The field is converging with systems design: hardware, kernels, and infrastructure choices are becoming as important as model architecture itself.
Summaries like this, in your inbox every morning.
Inference engineering has crystallized as its own discipline only within the past three years, yet it now sits at the center of AI infrastructure competition. The field addresses a fundamentally different question than model training: given weights that work on paper, how do you deploy them fast, reliably, and affordably? Baseten's $13 billion raise reflects investor recognition that this problem is now as economically important as training itself.
The breakthrough discoveries cited in the article reveal why inference engineering matters. Quantization—reducing model precision—was long thought to destroy quality; yet Philip Kiely and Ali Taha's work shows that errors introduced by quantizing different layers can interfere constructively, yielding 20% throughput gains while matching benchmark scores. This insight means that companies like Nvidia (mentioned in the article as having previously sacrificed speed or intelligence) were leaving massive optimization opportunities on the table. The convergence of these findings with Baseten's ability to graft Kimi's vision encoder onto GLM-5.2 without retraining demonstrates that inference engineering is becoming a systems discipline: optimizing hardware, kernels, cache reuse, and routing simultaneously.
The article suggests that inference optimization can still deliver gains of 20%, 100%, or even 200%—a range that underscores how immature the field remains and how valuable continued advances are. Baseten's ability to support new open models on day zero (as shown by their GLM-5.2 work) indicates that inference engineering is becoming a service moat: whoever masters these optimizations fastest can make open models competitive with proprietary ones.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
