Major audit reveals LoCoMo long-context benchmark has significant flaws with 6.4% corrupted answers and an LLM judge that accepts up to 63% of wrong responses.

Researchers identified 99 score-corrupting errors across 1,540 questions (6.4%) in LoCoMo, a widely-cited ACL 2024 long-term memory benchmark still receiving submissions as of March 2026

Error types include hallucinated facts in the answer key, incorrect temporal reasoning, and speaker attribution mistakes, such as specifying 'Ferrari 488 GTB' when only generic descriptions exist in source materials

The LLM judge evaluating answers accepts up to 63% of intentionally wrong responses, raising concerns about evaluation reliability

Alternative benchmark LongMemEval-S is limited as a true memory test since each question's corpus fits entirely within modern context windows