Apple researchers map alignment methods for multimodal AI models

Apple's Machine Learning research team has published a comprehensive study on preference alignment in multimodal large language models, authored by Elmira Amirloo, Jean-Philippe Fauconnier, Christoph Roesmann, Christian Kerl, Rinu Boney, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, and Peter Grasch. The paper examines how to improve MLLMs—AI systems that understand both text and images—by aligning them to human preferences.

The core problem the researchers address is hallucination in multimodal contexts. Unlike text-only models, MLLMs can hallucinate in two ways: by stating incorrect facts, or by generating responses that contradict the visual information presented. The team's objective is to encourage these models to ground their outputs more firmly in actual image content. While recent work has introduced preference datasets and tested alignment methods like Direct Preference Optimization (DPO) and Proximal Policy Optimization (PPO), the lack of standardization across these studies has made it unclear which specific components—datasets, base models, or algorithms—drive the reported improvements.

The researchers take a systematic approach, independently analyzing each aspect of preference alignment. They categorize alignment algorithms into two broad groups: offline methods (exemplified by DPO) and online methods (exemplified by online-DPO). A key finding is that combining offline and online approaches can enhance model performance in certain scenarios, suggesting that no single method is universally optimal. The team also reviews existing multimodal preference datasets in detail, examining how construction choices affect downstream model quality.

The study's main contribution is the introduction of Bias-Driven Hallucination Sampling (BDHS), a new technique for generating multimodal preference data. Unlike prior approaches, BDHS requires neither additional human annotation nor external models, yet achieves performance competitive with previously published alignment work when tested across multiple benchmarks. This efficiency gain is significant because data preparation is often a major cost and resource bottleneck in alignment research, and removing those constraints may accelerate progress in making multimodal AI systems more reliable and aligned with human expectations.