OpenAI's Habitat cuts read latency to 10 ms for 1B users

OpenAI's Habitat began as a simple client library and database and grew into a globally distributed storage platform. In Part II of its series, OpenAI explains that it initially relied on Azure Cosmos DB's built-in primitives but ran into problems with partition key collisions, noisy neighbors, and weak partitions as data grew. The company then tried single-tenant containers for isolation, but that approach made capacity planning a burden for every product team. OpenAI ultimately shifted back to a multi-tenant design after working with Azure to add hot key throttling, request tiering, and an improved serverless mode. These changes eliminated the need for manual provisioning while preserving isolation between workloads.

OpenAI also describes how its relationship with Azure evolved. As one of the largest users of Azure Cosmos DB, OpenAI found that managed services become less managed at its scale, and outages in late 2024 exposed gaps in failover testing. The two teams now jointly monitor every layer from service down to physical storage, share dashboards, and can page each other directly without going through support tiers. This co-engineering was critical to stabilizing the system.

On performance, OpenAI focused on read-heavy workloads, which represent over 90% of Habitat requests. It replicated small latency-sensitive datasets widely, integrated a distributed cache using Azure Cosmos DB's change feed for invalidation, and rewrote the Habitat service in Rust. Together these changes brought median global read latency down from roughly 120 ms to roughly 10 ms. OpenAI says it plans to address the remaining slowest requests by placing data closer to where it will be accessed as more capacity comes online.