Transformer limits spur 4 new LLM approaches

The transformer, introduced in 2017, solved a critical challenge in processing sequential data: it allowed AI models to weigh the importance of different words in a long passage simultaneously, rather than processing words one at a time. This architectural innovation became so effective that it is now embedded in every major commercial large language model. However, that same mechanism—comparing every word to every other word in a document—creates a computational cost that scales poorly. As text length grows, the number of comparisons multiplies, forcing models to consume more and more electricity to produce answers. This efficiency problem has become acute as applications demand longer context windows (the amount of prior text a model can "remember" when generating answers). The article frames this as the "transformer problem," suggesting that the constraint is now inhibiting further progress. Four startups have identified this gap and are pursuing alternative directions, each betting that a different technical path can break the transformer's dominance without sacrificing the quality users have come to expect from modern LLMs.