Agent memory beats RAG 15/16 to 13/16 in small test

The paper behind this test, "Memory Tax: Dissecting the Net Value of Agent Memory," compares 10 agent memory systems against a naive RAG that searches conversation history directly, and reports that many memory systems fall below RAG's raw score. It splits the effect into a benefit — memory answering what RAG misses — and a tax — memory answering RAG-answerable questions wrongly. The author was drawn to the explanation that summary memory discards details while naive search can still pick them up if keywords match.

The author could not find the paper's code and skipped its large benchmarks and multiple backbones because they would not fit a six-hour limit or require paid API keys. Instead, the author wrote two fictional multi-session logs — an internal dashboard redesign project with 37 turns and a family trip plan with 34 turns — seeded with one-off facts, then built a summary memory and a naive character-bigram TF-IDF retriever over the same logs. The summary memory kept current values and a change history for explicitly discussed decisions but dropped incidental details. Both conditions answered the same 16 questions, with answers fixed before either condition ran.

The result was 15/16 for memory and 13/16 for RAG, with 3 benefit cases and 1 tax case. The author notes this does not reproduce the paper's headline, and attributes the gap to the memory being far simpler than the 10 architectures tested and to the retriever lacking dense search or cross-encoder reranking — the paper reports RAG-Hybrid at roughly 70%. The author also flags that the two conversations and 16 questions carry no statistical significance, that Claude itself set the memory policy and answered under both conditions, and that the paper's specific architectures and backbones were not used.