New silicon architecture cuts LLM memory bottleneck by 99%

Researchers have introduced a new computational paradigm called the Persistent State Machine (PSM) designed to address what has become the central bottleneck in modern AI: the von Neumann memory wall. This term describes the growing gap between how fast processors can compute and how fast they can access data from memory. In large language models, especially during the inference phase where the model generates text, this data movement has become more costly in both energy and time than the actual mathematical operations.

The key innovation is architectural: instead of fetching data from memory to a central processor (the traditional model), ASMA broadcasts computation instructions to in-memory cells that remain stationary and perform state transitions locally. The paper defines PSM as a mathematical 6-tuple and proves that it contains Deterministic Finite Automata while achieving equivalence to Linear Bounded Automata under physical memory constraints, giving it PSPACE-complete expressive power. This theoretical grounding ensures the model is not merely a heuristic but a complete computational framework.

The practical implementation, ASMA (Active State-machine Memory Architecture), focuses on LLM KV-cache attention—a critical computation in large language models where the model attends to its past context. ASMA produces the full softmax(QK^T)V output, a core attention operation. When tested against an NVIDIA GeForce RTX 3090 GPU baseline, ASMA achieves dramatic reductions: system bus traffic drops by 99.47%, and net step energy falls by 99.44%. Most strikingly, under the most realistic and demanding scenario—extreme long-context multi-batch decoding with a batch size of 32 and context length of 131k tokens—ASMA achieves a 2,129.19x physical speedup. This speedup is not theoretical; it reflects actual hardware behavior in the exact regime where modern LLM inference spends the most time and energy.

The work is protected under Japanese Patent Application No. 2026-177318, indicating the authors view this as a potentially commercializable innovation. The results suggest that the path to more efficient AI inference may not lie in faster processors but in fundamentally rethinking how and where computation occurs relative to memory.

The von Neumann architecture, the foundation of modern computing, separates processing units from memory. For decades this separation worked acceptably, but large language models have exposed its fundamental limitation: during inference, especially on long contexts, the bottleneck is not computational power but the sheer bandwidth and energy cost of moving data between the CPU and memory. ASMA inverts this relationship by broadcasting instructions to memory cells that perform computations locally—a paradigm shift that treats stationary data as the primary computational substrate.

The mathematical foundation is rigorous. PSM is defined as a formal 6-tuple computational model that contains Deterministic Finite Automata and achieves equivalence to Linear Bounded Automata under physical memory constraints, giving it PSPACE-complete expressive power. This is not a heuristic optimization but a complete rethinking of how computation itself is organized. The 99.47% reduction in bus traffic and 99.44% reduction in energy consumption are not marginal improvements but structural changes to the energy profile of inference. The extreme speedup (2,129.19x) emerges specifically in the regime where LLMs spend most of their inference time: decoding with large batch sizes and long contexts, where data movement costs dominate.

No comments yet. Be the first to share your thoughts!