Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910121670475776 |
|---|---|
| author | Bhattacharyya, Apratim Xu, Bicheng Haresh, Sanjay Pourreza, Reza Liu, Litian Panchal, Sunny Madan, Pulkit Sigal, Leonid Memisevic, Roland |
| author_facet | Bhattacharyya, Apratim Xu, Bicheng Haresh, Sanjay Pourreza, Reza Liu, Litian Panchal, Sunny Madan, Pulkit Sigal, Leonid Memisevic, Roland |
| contents | Multi-modal Large Language Models (LLM) have advanced conversational abilities but struggle with providing live, interactive step-by-step guidance, a key capability for future AI assistants. Effective guidance requires not only delivering instructions but also detecting their successful execution, as well as identifying and alerting users to mistakes, all of which has to happen in real-time. This requires models that are not turn-based, but that can react asynchronously to a video stream, as well as video data showing users performing tasks including mistakes and their corrections. To this end, we introduce Qualcomm Interactive Cooking, a new benchmark and dataset built upon CaptainCook4D, which contains user mistakes during task execution. Our dataset and benchmark features densely annotated, timed instructions and feedback messages, specifically including mistake alerts precisely timestamped to their visual occurrence in the video. We evaluate state-of-the-art multi-modal LLMs on the Qualcomm Interactive Cooking benchmark and introduce LiveMamba, a streaming multi-modal LLM designed for interactive instructional guidance. This work provides the first dedicated benchmark and a strong baseline for developing and evaluating on live, situated coaching. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_21998 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance? Bhattacharyya, Apratim Xu, Bicheng Haresh, Sanjay Pourreza, Reza Liu, Litian Panchal, Sunny Madan, Pulkit Sigal, Leonid Memisevic, Roland Computer Vision and Pattern Recognition Multi-modal Large Language Models (LLM) have advanced conversational abilities but struggle with providing live, interactive step-by-step guidance, a key capability for future AI assistants. Effective guidance requires not only delivering instructions but also detecting their successful execution, as well as identifying and alerting users to mistakes, all of which has to happen in real-time. This requires models that are not turn-based, but that can react asynchronously to a video stream, as well as video data showing users performing tasks including mistakes and their corrections. To this end, we introduce Qualcomm Interactive Cooking, a new benchmark and dataset built upon CaptainCook4D, which contains user mistakes during task execution. Our dataset and benchmark features densely annotated, timed instructions and feedback messages, specifically including mistake alerts precisely timestamped to their visual occurrence in the video. We evaluate state-of-the-art multi-modal LLMs on the Qualcomm Interactive Cooking benchmark and introduce LiveMamba, a streaming multi-modal LLM designed for interactive instructional guidance. This work provides the first dedicated benchmark and a strong baseline for developing and evaluating on live, situated coaching. |
| title | Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance? |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.21998 |