Intel tests moving AI cache to system memory to ease GPU bottleneck

At the 2026 OCP APAC Summit in Taipei, Intel Platform Validation Engineer Jeff Ke presented test results demonstrating a potential solution to one of the most pressing challenges in large language model deployment: GPU memory bottlenecks. The key finding is that moving the key-value cache (KV-cache)—a critical data structure used during LLM inference—from GPU VRAM to system DRAM can raise serving throughput and enable a system to support more concurrent requests when VRAM becomes the limiting factor. The KV-cache is generated during the inference process and grows with each token an AI model produces; for applications serving many users simultaneously, it can quickly exhaust available GPU memory, forcing a choice between upgrading hardware or rejecting requests. By offloading this cache to system memory instead, Intel's tests indicate that deployments can maintain higher request concurrency even as GPU memory fills, allowing data centers to extract more value from existing GPUs. The presentation at OCP APAC suggests Intel is positioning this technique for broader industry adoption and validation across real-world deployment patterns.