IBM-backed llm-d hits version 1.0 for in-house AI serving

The llm-d project began last year as a collaboration among IBM, Red Hat, Google, CoreWeave and Nvidia, and is now hosted as a sandbox project by the Cloud Native Computing Foundation. Version 1.0 shifts the focus from getting models to serve at all to running them as production services, a step IBM says was missing for enterprise adoption.

The framework works with inference engines such as vLLM and SGLang, supplying distributed routing around model servers. IBM cited a recent agentic coding test in which the router captured 97% of available cache reuse, and described techniques that move cached information among GPU memory, CPU memory and storage when GPU capacity is constrained.

The economics matter more as agents make multiple rounds of reasoning, tool use and revision per assignment, IBM said, because reprocessing identical material each step consumes extra computing. Ruchir Puri of IBM Research linked open models and llm-d to an 11.8X efficiency advantage in computationally demanding work such as designing hardware for quantum error correction, and said version 1.0 extends those savings with operational readiness.