What is the core problem transformers face?

Transformers use dense attention, which compares every word with every other word via multiplication. Processing a 10,000-word document requires 50 million multiplications, making the computational cost grow rapidly as text length increases. This also limits how much information (context window) transformers can keep track of at once.

How much energy do LLMs consume today?

OpenAI is set to spend $50 billion on computing this year. The International Energy Agency predicts that total electricity consumed by data centers will double by 2030.

Which startups are building alternatives and what are their approaches?

Subquadratic uses sparse attention (calculating only some word pairings instead of all). Manifest AI uses power retention (rolling summaries of context). Liquid AI pairs liquid neural networks with transformers. Inception applies diffusion (from image generation) to generate whole text blocks at once. Pathway builds non-language-based representations like Dragon Hatchling.

Large Language ModelsMIT Technology Review AIAug 10, 2026Aug 10, 2026, 19:00 JST

Four startups challenge transformers, the engine of all major LLMs

Send on LINE

3 Key Points

What happened
MIT Technology Review examined a wave of startups building alternatives to transformers, the neural network architecture that powers every major language model today. The startups—Subquadratic, Manifest AI, Liquid AI, Inception, and Pathway—are developing sparse attention, power retention, liquid neural networks, diffusion-based generation, and non-language-based architectures to address transformers' core limitation: the computational cost explodes as text length grows.
Why it matters
Transformers require 50 million multiplications to process a 10,000-word document, driving enormous energy costs—OpenAI is set to spend $50 billion on computing this year, and the International Energy Agency predicts data center electricity consumption will double by 2030. Transformers also struggle to maintain large context windows and handle reasoning tasks efficiently, bottlenecks that become critical as LLMs tackle harder problems. These startups claim their approaches can make models faster, cheaper, and more efficient without sacrificing performance.
What to watch
Subquadratic claims its sparse attention model SubQ rivals mainstream LLMs on search and coding tasks, with thousands on its waitlist. Liquid AI's hybrid models match rivals four times larger and run on a $50 Raspberry Pi; the company has achieved almost 34 million downloads. Inception's diffusion-based Mercury 2 reportedly matches GPT-4 performance at 10 times faster speed. Pathway's Dragon Hatchling solved more than 97% of 250,000 hard sudoku puzzles, outperforming leading models from major labs.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Get news like this every morning, free

Context & Analysis

The transformer architecture, introduced by Google researchers in 2017 and described in the paper 'Attention Is All You Need,' has become the foundation of every major LLM on the market. Its ability to process long sequences of data, particularly text, made it revolutionary. However, as LLMs have grown in scale and capability—especially with the rise of reasoning models and larger context windows—transformers' core strength has become a bottleneck. The dense attention mechanism that made transformers powerful is fundamentally limited by its computational complexity, which scales with the square of input length.

This limitation has become acute precisely because of what LLMs are now asked to do: reasoning models require chain-of-thought scratch pads that expand the data an LLM must process, and next-generation systems (agents) need to ingest entire libraries of documents or code bases. The energy and cost implications are real—OpenAI's $50 billion annual computing spend and the International Energy Agency's projection of doubled data center electricity by 2030 underscore the urgency. The startups profiled represent four distinct philosophical approaches: making attention itself smarter and sparser, replacing attention with alternative mechanisms, using hybrid architectures that blend old and new neural network types, and borrowing generative techniques from other modalities. That multiple independent teams are pursuing fundamentally different solutions suggests the transformer successor is not yet obvious, but the search is well underway.

FAQ

What is the core problem transformers face?: Transformers use dense attention, which compares every word with every other word via multiplication. Processing a 10,000-word document requires 50 million multiplications, making the computational cost grow rapidly as text length increases. This also limits how much information (context window) transformers can keep track of at once.
How much energy do LLMs consume today?: OpenAI is set to spend $50 billion on computing this year. The International Energy Agency predicts that total electricity consumed by data centers will double by 2030.
Which startups are building alternatives and what are their approaches?: Subquadratic uses sparse attention (calculating only some word pairings instead of all). Manifest AI uses power retention (rolling summaries of context). Liquid AI pairs liquid neural networks with transformers. Inception applies diffusion (from image generation) to generate whole text blocks at once. Pathway builds non-language-based representations like Dragon Hatchling.

MIT Technology Review AIRead Original Article

Send on LINE

Get the latest Large Language Models news every morning

For example, today's edition would include:

Jensen Huang says AI safety is engineering — OpenAI cases say otherwiseYahoo Finance AI · 1h ago
Microsoft opens Copilot Managed Runtime preview inside Microsoft 365Publickey · 1h ago
OpenAI's Boris Power: 80 to 90 percent of research targets GPT 7 and beyondTHE DECODER · 4h ago

AI-summarized, only the topics you pick: one digest a day via Email, LINE, or Slack.

Subscribe free →

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.