AIToday
Large Language ModelsQiita 機械学習Published: Nov 17, 2026, 16:00 JST

Linear to KDA: one part at a time — Linear keeps old data at 0.99, DeltaNet erases it

Send on LINE
Linear to KDA: one part at a time — Linear keeps old data at 0.99, DeltaNet erases it

3 Key Points

  1. What happened

    A Qiita article traces Softmax Attention → Linear → DeltaNet → Gated DeltaNet → KDA as a history of adding parts one by one — write, read, then dilute — and in a hand-run test with 8 entries, Linear left old numbers at a=0.99 while DeltaNet (β=1) erased them (a=0.00).

  2. Why it matters

    The progression shows that each added component — write, then read, then dilute — changes how much old information survives an update, which is the core trade-off when memory is fixed-size.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

Linear attention is framed as a series of small, incremental additions rather than a single breakthrough. Softmax attention stores every past token and compares the query against all of them, so memory grows with sequence length. Linear attention replaces that with a fixed-size state that is updated by adding each new key-value pair, which cuts memory to a constant but means old information is never truly removed.

DeltaNet then adds a read step: before writing, it checks what the current key would retrieve and corrects only the difference between the old and new value. Gated DeltaNet adds a scalar gate that dilutes the entire state before the update, allowing rapid forgetting when context switches. KDA makes that gate a vector so that dilution can happen per channel, which the Kimi Linear team describes as addressing the coarse, head-level forgetting of Gated DeltaNet.

A hand-run experiment isolates the effect of these parts: with 8 entries and 3 overwrites, Linear leaves old numbers at 0.99, while DeltaNet with β=1 erases them completely. The author notes that Kimi Linear combines KDA with standard attention (MLA) in a 3:1 ratio, which may reflect an effort to balance what is summarized in a fixed-size state against what is kept in a full database, presented as the author's observation rather than a reported result.

FAQ
What is the difference between Linear and DeltaNet?
Linear only adds new key-value pairs to a fixed-size memory, so old numbers remain. DeltaNet reads the current value and corrects only the difference, so old entries can be overwritten.
What does KDA add on top of Gated DeltaNet?
Gated DeltaNet dilutes the entire memory by a scalar α_t, while KDA makes α_t a vector to dilute per channel, so different parts of the memory can be forgotten at different rates.
Is the experiment based on a trained model?
No, the numbers come from hand-set gates running the update equations, not from a trained model, averaged over 20 seeds.
Qiita 機械学習Read Original Article
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleClaude Code, Codex, SWE-2 each fail differently, $340/mo user finds