Zep builds Konig to handle millions of idle graphs

The article explains why Zep moved away from conventional graph databases for its agent memory workload. Traditional systems like Neo4j assume one large graph stays resident in memory, with every request passing through a Cypher parser and query optimizer. But agent memory stored as a knowledge graph produces one graph per user, team, and project, and a single enterprise customer can run millions of them, most cold at any given moment. That makes the cost follow provisioned memory rather than reads actually served.

Konig was built around this access pattern by tiering storage: hot graphs in RAM, recently used graphs on local NVMe, and idle graphs in object storage. Because each query reads exactly one graph, PageRank can run inline in milliseconds rather than as a batch job. The system also attaches encryption keys and retention rules to each graph, and keeps two in-memory layouts of every graph, converting between them on demand.

The blog also frames Konig within the wider LLM caching landscape, distinguishing four types: KV cache for attention tensors per request, prefix caching keyed by hash chains over token IDs, prompt caching as the provider's billed version at 0.1x read and 1.25x write rates, and semantic caching storing finished responses by cosine similarity. The first three are exact-match and correctness-neutral, while the semantic cache is fuzzy-match and can return wrong answers with a 200 status.