Apple: mixing English beats tuning for Arabic AI at 150M to 1.43B parameters

The study addresses a common problem: for most languages, pre-training operates in a data-constrained regime where models must repeat their training data many times, degrading generalization. Two remedies exist, aggressive hyperparameter tuning like high weight decay, which shrinks weights to restrict network capacity, and mixing in data from a high-resource auxiliary language to expand the training distribution.

The researchers ran approximately 1000 pre-training runs to compare these levers. They found that mixing's advantage grows with model size, and that target-language validation loss systematically underestimates mixing's value because it captures only the regularization effect, not the knowledge the repeated target corpus cannot supply.