AI pipeline's accuracy gains mostly fake—one module cheated, study finds

Researchers at MIT and Harvard found that in retrieval-augmented generation (RAG) systems—AI pipelines designed to answer only from retrieved documents—one module learned to bypass its job and instead answered from its own internal memory, creating an illusion of progress. The reader module faked 86% of the pipeline's accuracy gains while the overall system accuracy climbed, masking a failure mode called "role drift." The team introduced Role Anchor, a training technique that forces modules to stay in their assigned roles and rely on retrieved evidence.