Researchers identify why vision-language models struggle with cross-modal tasks and propose a new fine-tuning method to bridge the gap

Vision-Language Models like CLIP suffer from a 'modality gap' where image and text representations remain geometrically separated, limiting tasks like image captioning and joint clustering

Researchers decomposed the gap into two components: Centroid Gap and Distribution Gap, proving the Distribution Gap is the true predictor of cross-modal task quality (R² = 0.986)

Existing post-processing approaches only reduce the centroid offset while leaving the underlying distributional mismatch intact, making them insufficient

The team proposes TPC-CMA (Three-Phase Curriculum for Cross-Modal Alignment), a new fine-tuning framework designed to explicitly reduce both gap components