Stanford study: 44% of LLM research claims already expired

The study compares a 2023 finding by Walters and Wilder, in which GPT-3.5 fabricated 55 percent of its scientific references, with the same prompts sent to current models. GPT-6.1 Sol fabricated 0 percent and Claude Sonnet 5.5 came in at 1 percent, while Qwen3.7-Flash still fabricated 69 percent, showing that improvement is not uniform across developers.

The researchers also warn against reading such gains as an all-clear. On two-hop questions that require combining two known facts, Claude Sonnet 5.5 still fails in about 10 percent of cases where it correctly answers both individual questions, compared with 43 percent reported for ChatGPT in the original paper.

Results vary widely even within the same release window, and neither release date, provider, nor model size reliably predicts the pattern. Queen and Zou point to a post by Andrej Karpathy on how perception gaps about LLM behavior affect trust and usage, and recommend that papers document model, version, and date and publish prompts and analysis code so findings can be re-tested.