Apple researchers introduce VICIS to test AI models on visual concept inference

Apple researchers—Nick Stracke, Kolja Bauer, Josh Susskind, Miguel Angel Bautista Martin, and Björn Ommer—have introduced a new evaluation task and solution approach to address a specific weakness in vision-language models. The task, called Visual Concept Inference from Sets (VICIS), works as follows: given a small set of images that all share a visual concept and a separate query image, the model must generate new images that preserve the concept defined by the example set while remaining consistent with the query image. For instance, if shown several images of the same object rendered in a watercolor style, the model should be able to apply that watercolor treatment to a new object.

The researchers tested current state-of-the-art vision-language models on this task and found they struggle significantly. The models tend to either disregard the visual context provided by the example images entirely, or they fall back on biased default outputs. This reveals that despite their sophistication in following textual instructions, these models have limited ability to infer shared visual concepts from purely visual examples and apply those concepts to new scenarios.

To close this gap, the researchers developed a training framework and novel architecture designed specifically to learn visual concepts from image sets and extract concept-specific embeddings from query images. They evaluated their approach on both synthetic data and large-scale real-world datasets drawn from ImageNet and WordNet. The results demonstrate that their model produces more accurate and diverse outputs compared to existing approaches. Importantly, it also generalizes well to unseen concepts it has not encountered during training, as well as to different modalities such as sketch-based inputs, suggesting the learned representations capture visual concepts at an abstract level rather than memorizing specific examples.

Vision-language models have become central to modern AI applications because they can follow complex textual instructions. However, this strength in text-based reasoning masks a fundamental weakness: the ability to reason from purely visual examples. The VICIS task targets this gap directly. Rather than relying on language descriptions, a model must observe visual patterns across multiple example images and extract the unifying concept—then apply it consistently to generate new images.

The poor performance of current state-of-the-art models on VICIS reveals that visual concept learning from examples is not an emergent property of scaling or instruction-following capability. The models either discount the visual examples in favor of textual biases or generate outputs that lack coherence with the provided context. This suggests that the architecture and training dynamics of today's vision-language models are not well-suited to this type of visual reasoning, a challenge the researchers address through a new training framework and architecture design.

No comments yet. Be the first to share your thoughts!