Researchers discover that language models larger than 1 billion parameters show detectable hallucination signals before generating any text, revealing a critical scale threshold for factuality awareness.

Study analyzed 7 autoregressive transformers ranging from 117M to 7B parameters to understand when LLMs decide to hallucinate

Models below 400M parameters showed chance-level accuracy (AUC 0.48-0.67) with no reliable factuality signals across generation positions

A qualitative phase transition occurs above ~1B parameters where hallucination-indicative representations peak at position zero—before any tokens are generated

Research used three fact-based datasets (TriviaQA, Simple Facts, Biography) with 552 labeled examples to track internal representations across model scales