Byte Transformers beat subword models as scale grows

The research challenges the prevailing assumption that processing raw bytes is computationally inefficient because sequences are longer. The authors argue that the extra sequence length is a feature, not a defect, providing additional computation under the same parameter budget. Using token-superposition training and hash embeddings, byte Transformers consistently hit lower optimal loss than subword models at matched parameter counts. The study also found that byte Transformers implicitly develop local text abstractions similar to external tokenizers, with selective layer constraints preserving performance. These learned structures create non-uniform generation difficulty, which can be exploited for speculative decoding to accept 3.4× more tokens than subword models. The findings suggest that sequence length, learned abstraction, and computation allocation are interconnected dimensions for future language-model design.