AI memory demand spreads beyond HBM, says Bernstein

The article highlights a shift in AI's memory demands, moving beyond the initial focus on high-bandwidth memory (HBM) to include conventional DRAM, NAND flash, and storage. This is driven by the different requirements of AI workloads—training, inference, retrieval-augmented generation, and agentic AI—each placing unique pressures on the memory stack. For instance, while training is compute-intensive and relies heavily on HBM, the decode stage of inference is memory-bound, with KV cache sizes growing with context length and concurrent users. This could limit service capacity, a key concern for AI providers.

The emergence of new memory tiers, such as CXL memory and Nvidia's "Storage Next," reflects an industry effort to manage performance, capacity, and cost. However, technical challenges remain, particularly for high-bandwidth flash, which aims to bridge the gap between HBM's bandwidth and NAND's capacity. Bernstein's ratings suggest confidence in established memory manufacturers, but the dynamic nature of AI workloads means investment opportunities will depend on how these technologies evolve.