In summer 2017 Google researchers published "Attention Is All You Need," introducing the transformer neural-network architecture. Transformers proved especially effective at processing long sequences such as text, and nine years later they power every major large language model (LLM) on the market. Justin Dangel, cofounder and CEO of Subquadratic, says "the entire AI industry is built on transformers."
However, transformers are showing limits. Many recent advances—like models tailored for reasoning or handling very large inputs—are not clean extensions of transformer technology but workarounds for its fundamental shortcomings. Increasingly, researchers and engineers are asking what comes next: LLMs will remain important, but their construction may change. (MIT Technology Review labeled this next generation "LLMs+" in its list of AI priorities.)
The core problem: dense attention’s compute cost
Transformer strength comes from dense attention, which encodes the meaning of a text block by comparing each token with every other token. That yields accurate meaning representations but requires computation that scales quadratically with input length. For example, a 10,000-word document might require roughly 50 million multiplications. This explains in large part why LLMs consume so much compute.
The costs are substantial. According to Greg Brockman, president of OpenAI, the company will spend about $50 billion on computing this year. The International Energy Agency forecasts that electricity consumption by data centers could double by 2030.
Transformers also struggle to keep track of very large amounts of information simultaneously: their context windows can’t be expanded indefinitely. Yet to perform harder tasks, LLMs need to ingest larger data volumes—entire libraries, full codebases, or outputs from other LLMs. Reasoning models exacerbate this by writing and rereading intermediate notes (chains of thought), increasing the amount of data to manage.
Thus, the very mechanism that made transformers powerful has become a bottleneck.
Four approaches aiming to overcome transformer limits
Here are four approaches from startups that aim to make LLMs faster, more efficient, and in some cases able to perform new kinds of reasoning.
1) Rethinking attention: sparse attention and retention
One approach is to change how attention works. Sparse attention computes interactions for only some token pairings instead of all, greatly reducing computation. Many sparse attention schemes have been proposed, but historically none matched dense attention’s effectiveness at capturing meaning—until recently.
Subquadratic, based in Miami, claims to have developed the first sparse attention mechanism that rivals top mainstream LLMs on some tasks, including search and coding. Their model, SubQ, reportedly decides on the fly which words matter. The company says thousands have joined its waitlist and plans wider availability; some in the industry remain skeptical.
San Francisco’s Manifest AI tackles the problem differently by replacing attention with what it calls "power retention." This mechanism stores only the most relevant information for a task and maintains a rolling summary of the context window: as new information arrives, less relevant data is dropped. Where sparse attention discards many individual tokens but keeps a rough picture of everything seen, power retention seeks to keep a compact, task-focused summary.
Manifest AI says retention techniques have existed for a decade but that it updated them to build models that can compete with transformer-based LLMs. The company claims a transformer can be adapted into a power retention model with minimal retraining; it converted the open-source StarCoder into PowerCoder and released Brumby, which it says rivals some versions of Alibaba’s Qwen. Manifest AI positions power retention for tasks requiring huge amounts of data processing, from hours-long video analysis to agents that must stay on task for weeks.
2) Smaller, more flexible models: liquid neural networks
Liquid AI, an MIT spinout in Cambridge, Massachusetts, pairs transformers with liquid neural networks to produce "liquid foundation models" (LFMs). These models are much smaller and use less energy than typical LLMs. Liquid AI builds models for automakers such as Mercedes that run on tiny chips inside vehicles; its latest models can run on a $50 Raspberry Pi. The company offers models free to organizations with annual revenue under $10 million and reports nearly 34 million downloads.
Liquid neural networks were inspired by worm brains and extend earlier convolutional networks. Their key feature is a mechanism that lets a model adapt its behavior to new information during operation—i.e., they can learn on the fly—unlike transformers whose behavior is fixed after training. Liquid AI’s LFMs are hybrids: roughly 20% transformer and 80% liquid neural network. The company uses an internal AI designer to explore many neural-network combinations and find efficient architectures.
Liquid AI’s CEO Ramin Hasani argues transformers were only the beginning, pointing to biological brains that run on about 20 watts and suggesting there’s room for far more energy-efficient and innovative systems.
3) Generating text all at once: diffusion-based LLMs
Most LLMs generate output token by token, which mirrors human language but is inefficient for computers. It can be faster and cheaper to generate whole sentences or paragraphs in one pass. Inception, a Palo Alto startup, applies diffusion techniques—better known from image and video generation—to text.
Diffusion models are trained to transform random noise into structured outputs by working on all elements at once. In images, a model turns noisy pixels into a coherent picture; in text, Inception trains models to turn a random string of words into coherent sentences. These diffusion LLMs still use transformers to encode meaning, but by predicting many tokens simultaneously they achieve much greater speed and cost efficiency.
Applying diffusion to text required new math: unlike pixel values, discrete tokens don’t have obvious intermediate states (there’s no smooth transition between "cat" and "dog"). Stefano Ermon, Inception’s cofounder and CEO and a Stanford researcher, with colleagues worked out the math in 2024 and trained a diffusion model that matched GPT-2 performance while being ten times faster. Inception claims its latest model, Mercury 2, matches some 2023 GPT-4 variants in performance while again being ten times faster. Ermon frames the race as "intelligence per dollar," arguing speed and cost are decisive.
Google is also experimenting with diffusion LLMs (Diffusion Gemma), and Inception views such work as validation of the approach.
4) Moving beyond words: state-space models
Pathway, another Palo Alto startup, aims to free LLMs from the constraints of purely language-based representations. Its Dragon Hatchling model replaces transformer attention with a state-space mathematical structure that compresses information into more abstract representations rather than encoding it token-by-token.
Dragon Hatchling has had a standout result on a benchmark of over 250,000 very hard sudoku puzzles: it solved more than 97% of them, while several leading LLMs from top labs failed to solve any. Pathway’s argument is that some problems—sudoku, chess, certain mathematical reasoning—are poorly represented as sequences of words, and forcing reasoning into language can be constraining.
Pathway cofounder and CEO Zuzanna Stamirowska says transformers were an engineering convenience that became dominant, but breakthroughs can happen again. While a mainstream LLM could read about sudoku and write code to solve it, Pathway seeks models with capabilities beyond "book smarts," aiming at problem classes for which there is no existing recipe.
What this could mean going forward
Startups are exploring multiple paths: dynamic sparse attention, retention-based memory, hybrid liquid networks, diffusion for parallel generation, and state-space methods that abstract beyond sequences. Each approach targets lower compute and energy use, larger context handling, and new forms of reasoning. Some ideas already show promising results; others may fail. If one or more scale successfully, the economics and capabilities of LLMs could shift significantly in the coming years.
