New benchmark VSAS-Bench evaluates real-time streaming vision-language models with 18,000+ annotations to measure response timeliness and consistency.

VSAS-Bench introduces a specialized framework for assessing streaming vision-language models that continuously process video frames and generate real-time responses

The benchmark features over 18,000 temporally dense annotations across diverse input domains and task types, going beyond traditional single-turn video QA approaches

Evaluation metrics include proactiveness (measuring response timeliness) and consistency (capturing robustness over time), addressing limitations of offline-only VLM assessment

Standardized synchronous and asynchronous evaluation protocols enable comprehensive testing of visual streaming assistants in realistic deployment scenarios