Apple researchers propose GH-ESD for detecting vision model failures

Apple researchers have introduced GH-ESD (Grounded Hypothesis-Driven Error Slice Discovery), a new framework for identifying systematic failures in vision models, particularly those used for instance-level tasks such as object detection and image segmentation. The work addresses a key limitation in existing model evaluation: while models may perform well on average, they often fail systematically on specific, semantically coherent subsets of data—called error slices—revealing hidden robustness gaps. Existing approaches to discovering error slices model failures either as clusters in representation space or as combinations of predefined attributes. However, these formulations are insufficient for instance-level tasks, where failures frequently arise from contextual relationships and spatially grounded visual patterns that simple clustering cannot capture. GH-ESD reformulates slice discovery as a generate-and-verify framework. The system constructs relational failure hypotheses by combining LLM priors with grounded visual evidence, discovers hypothesis-specific slices at the instance level using Vision Language Models, and validates them through statistical trend analysis applied to instance-level errors. To support benchmarking and development, the team introduced GESD (Grounded Error Slice Dataset), a new benchmark that provides expert-defined and spatially grounded slices derived from real detection and segmentation failures. Extensive experiments demonstrated that GH-ESD consistently outperforms baseline methods: on the GESD benchmark for detection tasks, the framework achieved Precision@10 of 0.73, compared with 0.63 for baseline approaches—an improvement of 0.10. The method also supports segmentation scenarios. A key strength of GH-ESD is that the error slices it identifies are interpretable and actionable, enabling researchers and practitioners to understand specific model vulnerabilities and implement targeted improvements rather than relying on generic robustness techniques. The work was authored by Wei Zhang, Chaoqun Wang, Zixuan Guan, Ping Sheng Kao, Pengfei Zhao, Peng Wu, and Sifeng He at Apple.

Error detection in vision models remains a critical challenge because not all failures are created equal. Vision models often succeed on most data while breaking down systematically on specific subsets—a phenomenon that traditional evaluation metrics miss. Apple's GH-ESD addresses this by shifting from treating errors as random failures to identifying coherent patterns in when and where models fail. The framework leverages recent advances in large language models and vision-language models to generate hypotheses about failure modes grounded in actual visual patterns, then validates those hypotheses statistically over instance-level errors. This is particularly important for instance-level tasks (detection and segmentation) where failures are often tied to spatial relationships and contextual cues—patterns that simple clustering in representation space cannot capture. By making error discovery interpretable and spatially grounded, the work enables practitioners to move beyond generic robustness metrics toward targeted, actionable improvements.

No comments yet. Be the first to share your thoughts!