Ouro to 7.7T tokens: Looped Transformers, explained

The review comes from an author who admits he had filed Looped Transformers under 'just passing a Transformer through the same layers a few times' and stopped there. Following the papers, he argues, does not end at that description. Universal Transformers in 2018 already applied self-attention recursively along depth and added Adaptive Computation Time to stop computation per token — an early version of the current idea. What changed is the framing: Giannou et al. treated the loop not as a way to shrink a model but as a device for executing an algorithm, and Yang et al. tested whether ordinary training would recover repeated-algorithm behavior. The review also notes the input-injection trick added in that later work, where the original input is re-added at each loop so early information is not buried in the hidden state.

The later papers push scale and adaptivity. A 3.5B parameter model pre-trained on 800B tokens showed gains on math and coding when extra loops were run at inference, and Ouro carried the approach to 1.4B and 2.6B parameters on up to 7.7T tokens while pursuing adaptive computation, where harder inputs consume more depth.

The review's own open questions are about whether any of this is really thinking. A hidden state that changes across loops is not proof that one loop extracts, another reasons, and a third verifies. Until it is clear what each loop updates, whether loop count tracks problem difficulty, and whether a model can extrapolate past its training loop count, the outcome may hinge on whether stopping decisions can be learned stably — and for buyers comparing architectures, on whether the parameter savings survive contact with real inference costs.