Tensormesh and AMD let fewer GPUs run more AI models

On July 23, 2026, Tensormesh and AMD announced a partnership to optimize AI inference by allowing enterprises to run more models on fewer GPUs. The solution combines Tensormesh's LMCache (a KV cache management system) with AMD's virtual memory offering and GPU technology, specifically AMD Live Context Virtualization components. LMCache enables KV cache—temporary memory that AI models use during inference—to overflow from the GPU's high-bandwidth memory (HBM) onto a multi-tier storage hierarchy that includes DRAM, SSDs, and remote storage.

Testing was conducted on Dell servers equipped with 8x AMD MI355 GPUs and Dell storage. The results were substantial. On a 300 GB document set using the Kimi-K2.6 model, time-to-first-token (the time before the model produces its first response token) dropped from 3.4 seconds to under half a second—a nearly 7× improvement—by reusing cached KV data already stored in DRAM and NFS instead of recomputing it from scratch. Output throughput remained stable at approximately 48 tokens per second regardless of workload size, while unoptimized inference systems experienced a nearly 40% drop in throughput under the same conditions. Hardware model density doubled on the same infrastructure.

The economic case is clear: enterprises can reuse existing GPU infrastructure without buying new GPUs with larger memory, directly lowering costs and sidestepping the industry's current memory supply crisis. Customers can also maximize efficiency by reusing KV cache chunks stored for short-term memory virtualization later for prefix or non-prefix KV cache matching. Tensormesh CEO and LMCache co-creator Junchen Jiang stated: "Together, we're greatly enhancing the memory that inference engines can access for model weights and KV cache, using all of the memory resources on each node." AMD's Vice President of AI Software Anush Elangovan added: "We recognize Tensormesh and LMCache as KV cache management leaders. And we're thrilled to announce AMD's breakthrough in virtual GPU memory management, amplified by LMCache and Tensormesh." The collaboration builds on AMD's recent strategic investment in Tensormesh.

The collaboration addresses a structural problem in enterprise AI infrastructure: GPU memory (high-bandwidth memory, or HBM) is expensive and in short supply, yet inference workloads often need fast access to large amounts of temporary data (KV cache) generated during model computation. Rather than forcing customers to buy GPUs with more memory built in—the traditional approach that has contributed to the current memory supply crisis—Tensormesh and AMD's solution allows KV cache to overflow into cheaper, more abundant storage tiers: DRAM, SSDs, and remote storage. The key innovation is LMCache, which coordinates this multi-tier cache intelligently, allowing the system to reuse cached data for future requests (both prefix and non-prefix KV cache matching) rather than recomputing it from scratch.

The test results validate the economic trade-off: a nearly 7× speedup in time-to-first-token on large document sets, combined with doubled model density on the same hardware, means enterprises can consolidate workloads and reduce capital expenditure. The fact that output throughput remained constant even as load grew—while unoptimized systems degraded by 40%—suggests the solution scales well for real-world oversubscribed environments. AMD's recent strategic investment in Tensormesh and its simultaneous announcement of a virtual GPU memory management breakthrough indicate that this is not an isolated experiment but part of a broader AMD strategy to compete in the inference optimization market.

No comments yet. Be the first to share your thoughts!