Researchers find 'meta-tokens' that expose hidden model computation in Qwen3.6-27B

Researchers applied J-lens, a technique for finding meaningful activations in a model's internal state, to Qwen3.6-27B, a 27-billion-parameter language model. Their goal was to identify "meta-tokens"—tokens that surface computation the model performs that is not obvious from the text alone.

They found three striking examples. First, when the model reads ambiguous text, the token 什么意思 ("what does this mean") fires in the model's J-space representation. By steering this token away, they altered the model's interpretation of ambiguous inputs: "a boiled egg every morning is hard to beat" normally triggers a pun-catching response, but suppressing 什么意思 makes the model answer with nutrition facts instead. Second, on logical computation problems, the token "gcd" activates, pointing at the product/GCD algorithm the model is running. When they swapped the GCD vector from a value of 9 to 3, the model changed its answer for the LCM (least common multiple) of 27 and 90 from 270 to 810. Third, before the model hedges or lists multiple options, the token 大概率 ("most likely") fires; suppressing it forces the model to commit to a single answer—for example, turning the ambiguous statement "There are several logical places John could have gone" into a definite claim: "he went to the stationery store."

The researchers acknowledge that meta-tokens are challenging to discover with J-lens's current single-token constraint. However, they note that future methods capable of detecting and manipulating multi-token patterns could unlock much richer insights into the model's internal algorithms. This suggests a pathway from narrow probing of individual algorithmic steps toward a more complete map of how large language models execute reasoning.

Understanding how language models actually compute is a long-standing goal in AI research. Rather than treating the model as a black box that simply maps inputs to outputs, this work attempts to expose the internal mechanisms—the specific tokens that correspond to the model running particular algorithms. The use of J-lens, a tool for locating meaningful activations in a model's internal representation space, allowed the researchers to move beyond generic feature attribution and find tokens that correlate with concrete algorithmic behavior.

The practical value lies in steering: once you know which token corresponds to, say, a gcd calculation or a hedging behavior, you can manipulate that token's activation and change the output in predictable ways. This opens a path from interpretability (understanding what the model does) to controllability (directing what it does). However, the current method's single-token constraint limits how deeply researchers can explore—many algorithms may require multiple tokens working together. The body suggests that scaling J-lens to handle multi-token patterns is the natural next frontier.