
What happened
A Qiita article traces Softmax Attention → Linear → DeltaNet → Gated DeltaNet → KDA as a history of adding parts one by one — write, read, then dilute — and in a hand-run test with 8 entries, Linear left old numbers at a=0.99 while DeltaNet (β=1) erased them (a=0.00).
Why it matters
The progression shows that each added component — write, then read, then dilute — changes how much old information survives an update, which is the core trade-off when memory is fixed-size.
Summaries like this, every morning.
Linear attention is framed as a series of small, incremental additions rather than a single breakthrough. Softmax attention stores every past token and compares the query against all of them, so memory grows with sequence length. Linear attention replaces that with a fixed-size state that is updated by adding each new key-value pair, which cuts memory to a constant but means old information is never truly removed.
DeltaNet then adds a read step: before writing, it checks what the current key would retrieve and corrects only the difference between the old and new value. Gated DeltaNet adds a scalar gate that dilutes the entire state before the update, allowing rapid forgetting when context switches. KDA makes that gate a vector so that dilution can happen per channel, which the Kimi Linear team describes as addressing the coarse, head-level forgetting of Gated DeltaNet.
A hand-run experiment isolates the effect of these parts: with 8 entries and 3 overwrites, Linear leaves old numbers at 0.99, while DeltaNet with β=1 erases them completely. The author notes that Kimi Linear combines KDA with standard attention (MLA) in a 3:1 ratio, which may reflect an effort to balance what is summarized in a fixed-size state against what is kept in a full database, presented as the author's observation rather than a reported result.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
Innovation BASE Hokkaido updated AI:Promo's image editing with OpenAI's GPT-Image-2.6 Sunburst, announced Nove…

Softcreate's Yamaguchi published a four-phase guide for Microsoft 365 Copilot: prepare a daily-report prompt…

Google announced a new generation of intelligent eyewear co-developed with Gentle Monster, Warby Parker, and S…

Google Cloud's Threat Intelligence group (GTIG) said an LLM framework called exploit_agent, paired with a Russ…

A developer running Claude Code (Ultra, $220/month), Codex CLI (Pro, $100/month) and Devin's SWE-2 (Pro, $20/m…

The developer released mellow, which swaps chat for calls: you pick one of 20 AI friends, set in real cities…
