Gimlet Labs raises $300M for disaggregated AI inference

This round comes as AI developers face growing pressure to cut the cost and latency of running large models in production. Gimlet’s approach addresses a core inefficiency: an LLM is not a single uniform workload but a set of parts with very different hardware demands, so running the whole thing on one chip type can waste resources. By disaggregating the workflow and matching each piece to a specialized accelerator, the platform aims to make inference faster and cheaper—an outcome that could be significant for businesses that rely on AI at scale.

The company's method includes both the common PD disaggregation (splitting prefill and decode phases) and more granular splits, such as dividing the decode phase into smaller tasks or using a lightweight 'drafter' model alongside a larger one. Its software optimizes each module for its target chip using AI agents and a custom compiler. The funding will let Gimlet expand its serverless capacity by several hundred megawatts and move into custom hardware, indicating the company is betting on continued demand for specialized inference infrastructure.