PivotMerge framework merges multimodal AI models trained on different datasets to integrate their cross-modal alignment capabilities.

Researchers introduce PivotMerge, a post-alignment merging framework for cross-modal projectors (components that bridge visual and textual representations) in Multimodal Large Language Models. The approach is designed to integrate alignment capabilities from multiple expert models trained on heterogeneous data sources.

PivotMerge addresses two key challenges: cross-domain parameter interference (where parameter updates from different data distributions conflict) and layer-wise alignment contribution disparity (where different layers contribute unevenly to alignment). The framework uses Shared-space Decomposition and Filtering to disentangle shared alignment patterns from domain-specific variations, and Alignment-guided Layer-wise Merging to assign layer-specific weights based on differing contributions.

Experiments on CC12M-based post-alignment merging scenarios show PivotMerge consistently outperforms existing baselines across multiple multimodal benchmarks, demonstrating effectiveness and generalization ability.