Apple researchers propose integrated enlarge-and-prune LLM pipeline

What happened

Apple researchers, with collaborators from Georgia Institute of Technology and University of Texas at Austin, propose an integrated enlarge-and-prune pipeline for generative language model pretraining. The method combines enlarge model training, pruning, and recovery under a single cosine annealing learning rate schedule, and uses a novel iterative structured pruning method.

Why it matters

The study addresses whether pretraining an enlarged model is worth it even if it's never deployed, and how to optimize the pipeline for better pruned models. Experiments compressing 2.8B models to 1.3B with up to 2T tokens show the approach achieves superior performance of pruned models and provides insights into token efficiency.

What to watch

The paper suggests enlarged model pretraining can be efficiently integrated into pruning, potentially offering a more effective alternative to training target-size models from scratch within limited inference budgets. The full research paper is available at machinelearning.apple.com.