Researchers discover arithmetic generalization in minimal GPTs fails in three distinct stages, from layout shifts to carry operation misunderstanding.

A minimal GPT trained exhaustively on 2-digit addition fails to generalize to 3-digit problems despite having all required local digit transitions in training data

First failure stage is a 'layout barrier' where the model's absolute-position learning collapses under 3-digit layout shifts, only weakened by mixed-layout exposure

Second stage reveals the hundreds position is processed as a carry flag rather than a semantic digit, fixable through targeted carry probes that reverse logit margins

Third bottleneck is conditional recomposition, where high-conditioned tail data significantly outperforms matched control datasets in solving remaining failures