IBM-backed llm-d hits 1.0 with in-house AI controls

IBM, Red Hat and Google co-founded llm-d last year with CoreWeave and Nvidia, and the project now counts AMD, Cisco, Hugging Face, Intel and Mistral AI among its contributors. Version 1.0 shifts the focus to what happens after deployment: a production inference service must handle changing traffic, hardware failures, software updates and users competing for resources while maintaining acceptable response times.

The new controls are aimed at agentic workloads, where a single agent may issue many requests, pause to use an external tool and return for additional model calls. The framework's routing system directs requests toward replicas that already hold relevant cached context, and another capability moves cached information among GPU memory, CPU memory and storage when GPU capacity becomes constrained. IBM cited a recent agentic coding test in which the framework's router captured 97% of available cache reuse. In a separate demonstration, IBM and Red Hat said they ran 3,000 software-engineering agents concurrently on self-hosted infrastructure at a cost five to 10 times lower than serving the same open model through a hosted service.

Ruchir Puri, IBM Research's chief scientist, said open large language models and llm-d can greatly lower inference costs in computationally demanding problems such as designing hardware for quantum error correction, helping achieve a 11.8X efficiency advantage. The release is currently hosted as a sandbox project by the Cloud Native Computing Foundation.