Researchers create dataset to evaluate how well multimodal AI assistants can guide users through assembly tasks using instruction manuals and videos.

New dataset aligns instruction manuals with assembly videos to benchmark multimodal large language models (MLMs) on real-world procedural tasks

Study addresses the gap between current LLM capabilities and practical applications requiring both text and visual understanding

Researchers envision MLMs as real-time technical assistants integrated with VR/AR to provide guidance from the user's point of view

Work evaluates openly available multimodal models to assess their readiness for supporting complex, domain-specific assembly and procedural tasks