MTServe hierarchical cache system speeds up generative recommendation inference by up to 3.1× while managing GPU memory constraints

MTServe is a cache management system designed for generative recommendation models (AI systems that encode long user histories to make recommendations). It virtualizes GPU memory by using host RAM as backup storage to handle the large scale of individual user states.

The system includes a hybrid storage layout, an asynchronous data transfer pipeline, and a locality-driven replacement policy to bridge the performance gap between GPU and host memory tiers.

On both public and production datasets, MTServe delivers up to 3.1× speedup while maintaining near-perfect hit ratios (>98.5%).