Lightbits launches Inferra to boost AI inference performance

Lightbits Labs, known for inventing the NVMe over TCP storage protocol, is entering the AI inference market with Inferra, announced in March and now generally available. The engine manages KV cache across GPU high-bandwidth memory, DRAM, and NVMe storage, using predictive prefetching to stage data before the GPU needs it. This disaggregation addresses capacity and retrieval speed limits of GPU memory.

The company claims cache hit rates near 99.9% and positions Inferra as most beneficial for retrieval-augmented generation, AI agents, and heavily shared GPU services. Rasmusson compared the approach to CPU techniques from when processor speeds outpaced memory. Lightbits sees neocloud providers as early adopters due to their utilization and margin pressures.

Inferra's success hinges on whether the claimed benchmark results hold in real-world production. The company has pilots underway and will demonstrate at the AI Infra Summit next week, offering a concrete test of whether its staging predictions can deliver the promised latency improvements for operators.