AI model's learned token diverges from intended meaning

Researchers investigating how AI models learn and interpret new concepts conducted an experiment in which they trained a model to learn a new token—a neologism, following the methodology of Hewitt et al., but with a key difference. Rather than training the token on pre-labeled data, they trained it on data the model generated while being steered with a persona vector. This approach was designed to capture how the model itself responds to directional guidance. To understand what the model had learned, the researchers then asked it to perform two tasks: respond in the style of the newly learned token, and explain what that token means. The results showed that responses generated using the token exhibited substantially higher similarity to the original steering vector (larger projection values) compared to responses generated by direct steering alone. Critically, an independent LLM judge rated these token-generated responses as more coherent and more expressive of the intended trait. However, when asked to explain the token's meaning, the model's descriptions diverged from the intended persona. In one case, the model labeled the token as representing "dread" when researchers had intended it to represent "evil." In another, it identified "warmth" rather than "sycophancy." To test whether the mislabeling was incidental or structural, the researchers prompted the model to generate responses fitting the model's own category names but explicitly excluding the original trait—for instance, "dreadful but not evil." Remarkably, these responses still showed high similarity to the "evil" vector, despite being independently judged as barely evil at all. This disconnect suggests that while the model reliably encodes and executes the learned behavioral pattern, its linguistic explanation of that pattern operates on a different basis than its internal representation.

The research highlights a fundamental tension in how large language models learn and represent concepts. When researchers steer a model toward a desired persona using a vector, the model internalizes that pattern—the token trained on steered outputs reliably reproduces behavior aligned with the original vector. However, the model's ability to name or explain what it has learned lags behind its ability to execute the learned behavior. This gap is not merely a labeling error; it suggests that the model's internal representation space and its linguistic conceptualization operate according to different organizing principles. The fact that responses framed in the model's own mislabeled category ("dreadful but not evil") still activate the original vector while being judged as barely evil indicates that the model's behavioral substrate and its explanatory language are decoupled. For interpretability research, this poses a challenge: trusting a model to explain its own reasoning may be misleading, even when the model is responding coherently and the behavior it exhibits is measurable and reproducible.

No comments yet. Be the first to share your thoughts!