New Ukrainian Visual Word Sense Disambiguation benchmark challenges multilingual AI models, with CLIP-based baseline outperforming eight large language models.

Researchers created the first Visual-WSD benchmark for Ukrainian, following methodology from previous benchmarks in English, Italian, and Farsi

The task requires AI models to select the correct image representation of an ambiguous word from a set of ten images with minimal context

Eight multilingual and multimodal large language models were tested against a CLIP-based zero-shot baseline model

Benchmark data was collected semi-automatically and refined by domain experts to ensure quality

All tested models underperformed compared to the CLIP-based baseline, revealing challenges in cross-language visual disambiguation