No AI model passes 60% on visual perception benchmark

Moonshot AI, the team behind the Chinese AI assistant Kimi, has introduced PerceptionBench, a new benchmark designed to isolate and measure the visual perception capabilities of multimodal language models independently from reasoning and knowledge. The benchmark consists of 3,000 publicly released tasks drawn from an internal pool of over 17,000 verified questions; 60 percent were derived from actual attributed model errors, while 40 percent were reformulated using augmented images. Unlike standard evaluations that bundle perception, knowledge, and reasoning into a single assessment, every question in PerceptionBench can be answered by looking at the image alone, with no reasoning or outside knowledge required.

The authors built the benchmark by analyzing 42 existing open-source perception benchmarks and found they contain little overlap in their error profiles—each covers a different subset of visual weaknesses. Rather than imposing categories from theory, the team extracted them from real model errors, tracing each failure back to the earliest failed step in existing benchmarks. This produced ten "skill domains": Visual Relation, Counting, Attributes, Depth & 3D, Localization, Comparison, Fine-grained Recognition, Context Integration, OCR, and Hallucination. The tasks themselves appear deceptively simple on the surface: figuring out where a symbol sits on a clock face, counting flowers inside a red box, or distinguishing which of two pencil cups shows a gray-pink combo versus a solid pink with cartoon design.

When PerceptionBench was tested across 16 frontier models, no model reached 60 percent accuracy. GPT-5.6 Sol led with 59.7 percent, followed by Kimi K3 at 58.5 percent, Claude Fable 5 at 57.2 percent, Gemini 3.1 Pro at 56.2 percent, and GPT-5.5 at 55.8 percent. Open-source models trailed significantly: Qwen3.5-397B-A17B scored 47.5 percent, and GLM-4.6V scored 32.5 percent. The gap between the top five systems spans just under four percentage points. However, the category-level results reveal a more complex picture. Models with nearly identical overall scores diverge sharply on individual skills. Across all models tested, "hallucination" is the weakest domain—the ability to recognize when an object does not exist and answer "zero." GPT-5.6 Sol, despite leading overall, scored only 26.9 percent on hallucination, while Gemini 3.5 Flash, a weaker performer overall, ranked among the best at 50.6 percent.

The authors argue that many failures typically attributed to "reasoning errors" in multimodal models actually originate at the perception stage. When a model fails a multi-step task, the first step—correctly reading the image—has often already failed. PerceptionBench makes it possible to isolate and pinpoint which specific visual ability is failing. The dataset and evaluation code are available on GitHub. This work is part of a broader research effort: the same team previously released WorldVQA, a benchmark that separates object recognition from reasoning, where Gemini 3 Pro achieved 47.4 percent—below 50 percent—and all models systematically overestimated their own confidence. Another parallel study involving Chinese institutions and Moonshot AI used the BabyVision benchmark to show that frontier models fail at basic visual tasks tied to early childhood development, such as tracing lines or counting hidden blocks; Gemini 3 Pro scored 49.7 percent while humans achieved 94.1 percent. Researchers attribute this gap to a verbalization bottleneck where visual information is translated into language and loses fidelity. Meanwhile, Kimi K3, Moonshot AI's open-source offering, has closed the gap to Claude Fable 5 and GPT-5.6 Sol to within a few points on general benchmarks, though it still lags in specialized areas like offensive cybersecurity and complex math. On visual perception as measured by PerceptionBench, K3 now performs on par with its Western rivals.