Researchers discover Vision-Language Models suddenly fail at simple visual tasks, revealing a critical gap between image encoding and actual understanding.

Grid2Matrix benchmark tests VLMs' ability to read color grids and convert them to numbered matrices, exposing failures in exhaustive visual detail capture

VLMs show sharp, early collapse rather than gradual degradation, failing surprisingly on small grids despite excelling on standard multimodal benchmarks

Visual encoders preserve substantially more grid information than end-to-end model outputs, suggesting the failure occurs in later processing stages rather than image encoding

The controlled benchmark isolates visual complexity from semantic reasoning by varying only grid size and color count, minimizing confounding variables