OptiMer reduces the cost of optimizing training data ratios for continual pre-training by 15-35 times using post-hoc Bayesian optimization instead of expensive upfront tuning.

OptiMer trains separate models per dataset, then extracts distribution vectors representing parameter shifts and optimizes their composition weights after training rather than before

Tested on Gemma 3 27B across multiple languages (Japanese, Chinese) and domains (Math, Code), OptiMer outperforms traditional data mixture and model averaging approaches

The optimized weights discovered by OptiMer can be interpreted as optimal data mixture ratios, which when used in retraining further improves continual pre-training performance

Solves the problem of data mixture ratios being expensive hyperparameters that must be fixed before training starts and waste weeks of compute if chosen suboptimally