Append-only memory beats mutation; Raw-RAG only ties at 83.33%

The original paper, "Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought" (arXiv:2610.30755), argued that a simple raw-log retrieval baseline outperformed seven complex memory methods in six of seven model settings. This experiment set out to test that claim in miniature, with three fictional conversation logs where facts like city, job, and pet change over time. The immediate result was a clear split: the destructive part of the paper reproduced, but the headline advantage did not.

The turning point came from a follow-up test on a local GPU. With only TF-IDF, Raw-RAG scored 29.17% overall, well behind structured memory. Swapping in dense embeddings brought it to 62.50%, and adding a cross-encoder reranker pushed it to 83.33% — identical to AppendOnlyMemory. That suggests the earlier gap was largely an artifact of weak retrieval rather than a property of raw logs.

The author is careful to note the limits. Two of the four remaining misses were shared with AppendOnlyMemory, traced to expressions the simple regex extraction never captured. Verifying whether raw-RAG truly surpasses structured memory would likely require the paper's long-conversation benchmarks and a reader LLM to synthesize answers, which were not available here.