Vast: Alon Horev says AI agent memory moving to tiered storage

Vast's argument rests on a shift in how AI inference is expected to work. Horev described sessions that hold a KV cache in GPU memory and then go idle when the agent compiles code, tests software, or pauses for a person. Offloading a session to storage keeps that memory wall from forcing a repeat calculation, he said. The tiering order he outlined runs from GPU memory to CPU memory on the same machine to persistent media holding petabytes of KV cache, with Nvidia Corp.'s Dynamo software orchestrating the process.

Horev framed this as treating inference as a distributed problem across a fleet of machines. Moving a session from a busy GPU to a less busy one, and reading KV cache either over the network or from Vast, gives operators more optionality and more optimized scheduling, he said. That matters differently from ordinary context handling, he argued, because enterprise agents also need shared knowledge that persists across sessions, including long-term memory of past conversations.

He also tied agent memory to governance and training. Companies running thousands of agents that handle sensitive data and act for customers need to record everything those agents do and retain it for a set period, he said, and Vast has launched a confidential computing service for sensitive workloads. The stakes appear to hinge on whether enterprises adopt this tiered model at fleet scale and whether persistent KV cache proves to be as valuable as Horev suggests. Horev's claim that agent conversations are "gold" for fine-tuning, training, or purpose-built models is his view, not a stated benchmark.