Apple researchers benchmark long-video AI summaries with temporal grounding

Apple researchers led by Alkesh Patel, Melis Ozyildirim, Ying-Chang Cheng, and Ganesh Nagarajan have released a new benchmark designed to expose gaps in how current AI models summarize long videos. The problem is precise: multimodal large language models (systems that ingest both visual and textual information) struggle to maintain temporal fidelity—the accurate tracking of when events occur—over extended video durations, and the summaries they produce often lack both semantic and temporal grounding.

The LVSum benchmark addresses this by providing 72 diverse videos drawn from 13 different domains, each averaging 16 minutes in length. Critically, each video is paired with up to 10 human-written summaries that explicitly include temporal references—allowing researchers to measure not just whether an AI captured the right content, but whether it correctly identified when events happened. To evaluate model performance, the team employed both newly developed LLM-based metrics (measuring content relevance and what they call "modality coherence"—how well the AI balances information from different sources like audio and video) and conventional automatic evaluation methods.

When testing leading proprietary and open-source multimodal models against this benchmark, three findings stood out. First, transcripts—the text of what people say or narrate in the video—contributed substantially more to summarization quality than visual frames alone, suggesting models may be relying on text as a shortcut rather than genuinely reasoning about the visual narrative. Second, a significant performance gap remained between summaries written by humans and those generated by AI, indicating the task remains well beyond current system capabilities. Third, and most revealing for temporal reasoning, the models exhibited systematic weaknesses in three areas: temporal grounding (correctly identifying when things happened), instruction adherence (following user requests), and cross-modal coherence (integrating information from video, audio, and text consistently).

Long-form video summarization has emerged as a demanding test of multimodal AI capabilities. While large language models trained on vision and language data have shown impressive progress in static image understanding, extending that capability to video—where temporal sequence and event ordering matter—remains unsolved. Apple's benchmark directly addresses this gap by building evaluation infrastructure that captures what human annotators consider important: not just what happens in a video, but when it happens and how events relate across time.

The finding that transcripts outweigh visual frames challenges a common assumption in multimodal AI design. This suggests that current models may be taking shortcuts around genuine temporal reasoning, using text summaries of dialogue or narration as a substitute for understanding the visual evolution of events. The systematic weaknesses in temporal grounding and instruction adherence indicate that improvements will require explicit training objectives—not simply scaling existing architectures. For practitioners, these results signal that production video summarization systems today likely inherit these same biases and limitations.

No comments yet. Be the first to share your thoughts!