Researchers discover that vision-language models struggle to adopt alternative rule interpretations, preferring familiar semantic defaults even when explicitly instructed otherwise.

Study introduces VLM-Fix benchmark using four abstract strategy games to test whether 14 major VLMs can switch between standard and inverse rule formulations on identical board states

Consistent accuracy gap found across all tested models, with significantly better performance under standard rules, indicating a 'semantic fixation' behavior

Neutral prompt aliases effectively reduce the inverse-rule performance gap, while semantically loaded aliases reopen it, suggesting the bias is driven by semantic priors rather than perception errors

Models trained on single rule sets show strong same-rule transfer but poor opposite-rule transfer, revealing that post-training reinforces rule-specific associations rather than building flexible rule understanding