Four startups challenge transformers, the engine of all major LLMs

In the summer of 2017, Google researchers published 'Attention Is All You Need,' introducing the transformer architecture. This neural network proved exceptionally effective at processing long sequences of text, and nine years later, transformers power every major large language model in the market. Justin Dangel, cofounder and CEO of Subquadratic, calls them 'one of the most important innovations in the history of computer science.'

Yet transformers are beginning to show limitations. The mechanism at their heart, called dense attention, works by comparing every token (word or part of a word) in a text with every other token via multiplication. This encodes meaning with remarkable accuracy but becomes computationally expensive as text grows. A 10,000-word document requires 50 million multiplications. The costs are staggering: OpenAI is set to spend $50 billion on computing this year, according to president Greg Brockman. The International Energy Agency predicts data center electricity consumption will double by 2030. Beyond cost, transformers struggle to maintain large context windows—the amount of information they can track simultaneously—a critical limitation for reasoning models (which use chain-of-thought scratch pads) and agents (which need to process output from other LLMs or entire code bases).

Four startups are pursuing distinct solutions. Subquadratic, based in Miami, has developed SubQ, a sparse attention model that calculates relationships between only some word pairings instead of all of them. The company claims it rivals top mainstream LLMs on search and coding tasks, though skepticism persists in the industry. Thousands have signed up for its waitlist, and the company plans wider availability soon. Manifest AI, based in San Francisco, takes a different approach: power retention, which stores only the most relevant information and maintains a rolling summary of context rather than tracking everything. The company claims to have updated a decade-old technique and demonstrated it by converting StarCoder (an open-source coding LLM) into PowerCoder and releasing Brumby, which it says rivals versions of Alibaba's Qwen model. Cofounder and CTO Carles Gelada identified applications in analyzing hours-long videos and building agents that can stay on task for weeks.

Liquid AI, an MIT spinout based in Cambridge, Massachusetts, pairs transformers with liquid neural networks to build what CEO Ramin Hasani calls liquid foundation models (LFMs). These models are far smaller and more energy-efficient, running on small chips in vehicles and even on a $50 Raspberry Pi. They are available free to organizations with annual revenue under $10 million and have been downloaded almost 34 million times. Liquid neural networks, inspired by worm brains and descended from convolutional networks, adapt their behavior to new information as they process it—a capability transformers lack once training is complete. Liquid AI's hybrid models use a 20% transformer, 80% liquid neural network ratio and match the performance of rivals four times larger, including Qwen and Google's Gemma.

Inception, based in Palo Alto, applies diffusion—the technique that powers most image and video generation—to text. Rather than generating one word at a time, diffusion LLMs produce whole blocks of text at once. In 2024, cofounder and CEO Stefano Ermon (also a Stanford researcher) and colleagues solved the mathematical challenge of applying diffusion to text, training a model that matched GPT-2 performance at 10 times faster speed. Today, Inception claims Mercury 2 performs as well as some OpenAI GPT-4 models (released in 2023) but again 10 times faster. Ermon says the only metrics that matter are speed and cost: 'Ultimately, the currency is going to be intelligence per dollar.' Google is also experimenting with diffusion and has built a prototype called Diffusion Gemma.

Pathway, also based in Palo Alto, pursues perhaps the most radical approach: freeing LLMs from language constraints. Its Dragon Hatchling model (named after dragons in Terry Pratchett's Color of Magic) achieved a high score solving hard sudoku puzzles, beating more than 97% of 250,000 puzzles—a benchmark where several leading models from top labs failed to solve any. The point Pathway wants to make is that language-free representations may unlock capabilities transformers cannot reach.