Apple's CapQuiz grades video captions by quiz answers

Video captioning has long been judged by comparing a model's generated text against a ground-truth reference. The Apple researchers behind CapQuiz argue this is a poor fit for video, because a single video can be described in many valid ways. Two captions can both be accurate while emphasizing different visual details, and under reference-matching that legitimate difference shows up as an error. The authors also note that existing assessments tend to be one-dimensional, without a fine-grained breakdown of caption quality.

Their alternative reframes the question from "does this caption look like the reference?" to "does this caption carry the information that matters?" That is what they call information fidelity: a good caption should cover salient visual content while remaining strictly factual. CapQuiz operationalizes this by turning each video into human-verified multiple-choice questions, spanning a hierarchical taxonomy of 10 question types across Descriptive and Inferential categories and 24 video domains. CapF1 then separates the score into two parts, CapP for factuality and CapR for coverage, so the two dimensions can be read independently.

The stated payoff is empirical: across extensive experiments, CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance. Whether that translates into a durable standard is likely to hinge on adoption outside this research setting, and on whether teams find the question-based signal easier to act on than a single similarity score.