Agentic AI shifts focus from training to storage, memory, power efficiency

At RAISE Summit, a series of conversations among infrastructure leaders showed how agentic inference is redefining the AI data center. Unlike the previous phase of AI, which centered on training large models at scale, agentic inference—where AI agents run continuous, long-duration tasks—is creating new demands on storage, power, and system architecture.

Greg Matson, senior vice president and head of marketing and products at Solidigm (a trademark of SK Hynix NAND Product Solutions Corp.), explained that storage has moved from the periphery to the critical path. "It started a couple of years ago with training, where the need for high-capacity, high-performance storage very adjacent to the GPUs was all of a sudden center stage," Matson told theCUBE. "But now, as we go from last year to this year, inference phase into agentic inference, it's exploding even more. Storage is actually a whole new storage tier that's being created to extend the memory for the system." Solidigm is testing this approach through its AI Central Lab, which runs actual agentic workloads across accelerator hardware and partner software to understand how storage behaves in production environments rather than in isolation.

The specialization across the AI stack is also evident in how individual companies are evolving their product strategies. AMD's Mark Papermaster noted that enterprises are no longer optimizing for single bespoke tasks but for "whole processes" that require different computing engines working together at scale. AMD's ROCm software stack is designed to provide a consistent layer across data center clusters, edge deployments, and AI-enabled PCs. Tensordyne is tackling power constraints through a different innovation: its Napier inference chip uses a proprietary Pareto logarithmic number system that replaces multiplications with additions, reducing reliance on large, power-intensive multiplier circuits. According to co-founder Gilles Backhus, a 72-chip Napier pod draws 30 kilowatts, compared with 150 kilowatts for a comparable Nvidia system. "Our logarithmic math — it's completely under the hood," Backhus explained. "From a user point of view, from a [software development kit] point of view, you don't even notice it. It just looks like normal floating-point math. It's just that the engine under the hood is more efficient." Meanwhile, d-Matrix is pioneering heterogeneous inference by pairing its Corsair accelerators with Nvidia Hopper and Blackwell GPUs to serve different requirements: compute-heavy prefill and latency-sensitive token generation. As d-Matrix co-founder Sudeep Bhoja noted, "Low latency is the name of the game today. Agents are running for a long time; users don't want to wait."

Beyond hardware, two additional constraints are reshaping how enterprises deploy agentic systems. Argentum AI CEO Andrew Sobko identified capital as "the biggest bottleneck in the focus on the speed of deployment." Argentum AI's model secures customers and contracted revenue before committing capital to construction, treating financing as an integrated part of the deployment stack. Simultaneously, data sovereignty is moving from a compliance concern into infrastructure architecture. Agentcy Labs CEO Amit Eyal Govrin and Neo4j CTO Philip Rathle described sovereignty as spanning territorial, operational, stack, legal, and unit economics concerns. "Sovereignty is exerting agency and control over your AI," Govrin said. "You have to be free and clear of state, economic and threat actors overtaking any level of control over your stack." Neo4j is advancing knowledge graphs as a control mechanism—allowing some decisions to run deterministically rather than relying entirely on probabilistic large language models. As Rathle explained, "Having the capacity to do AI with a full brain, both hemispheres, is ultra important. LLMs are spontaneous, creative — they make mistakes, you don't know why. Having the graph as the left brain to the LLM right brain is really at the core of where graphs fit in."

The article documents a fundamental shift in how enterprises design AI infrastructure. The race to scale training—which dominated the past two years—has given way to a phase where inference, particularly agentic inference (longer-running AI agent tasks), is reshaping priorities. Agentic workloads are exposing new bottlenecks that were previously secondary: storage capacity and bandwidth, power efficiency, and the need for heterogeneous computing (specialized accelerators working alongside GPUs rather than replacing them).

This shift is not merely a technical optimization but a strategic reorientation. As Greg Matson explained, hyperscalers are now treating storage as an active extension of GPU memory, not an isolated component. The imperative is to keep GPUs "humming 100% of the time generating tokens," because idle GPU time wastes the largest capital investment. This logic has also accelerated the adoption of power-efficient architectures—Tensordyne's use of logarithmic math to replace multiplications with additions is a concrete example of rethinking silicon design for the constraints of continuous inference rather than batch training.

Beyond hardware, the article reveals that capital and governance have become infrastructure concerns. Argentum AI's financing-first model and the emphasis on data sovereignty through knowledge graphs indicate that enterprises building agentic systems are no longer purchasing isolated components but assembling integrated stacks that account for speed of deployment, cost of capital, and control over proprietary data. This reframing—treating capital, sovereignty, and architecture as co-equal challenges—signals a maturation of the agentic AI market from proof-of-concept to production scale.

No comments yet. Be the first to share your thoughts!