Researchers develop KALAVAI, a method to predictably combine independently fine-tuned AI specialists into a superior fusion model using a lightweight MoE router.

The approach lets multiple people independently fine-tune copies of a base model on different domains/languages with no communication, then combines them with a lightweight Mixture of Experts router trained in ~500 steps

Testing on Pythia models (410M to 6.9B parameters) shows consistent gains of 6.5-8% over the best individual specialist

A simple linear formula (R² = 0.856) can predict fusion effectiveness from specialist divergence before training begins: gain = 0.82 × divergence − 2.72

Code, paper, and project page are publicly available on GitHub and arXiv, with particularly promising cross-lingual fusion results