Robot Confirmation Generation and Action Planning Using Long-context Q-Former Integrated with Multimodal LLM
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912723141394432 |
|---|---|
| author | Hori, Chiori Masuyama, Yoshiki Jain, Siddarth Corcodel, Radu Jha, Devesh Romeres, Diego Roux, Jonathan Le |
| author_facet | Hori, Chiori Masuyama, Yoshiki Jain, Siddarth Corcodel, Radu Jha, Devesh Romeres, Diego Roux, Jonathan Le |
| contents | Human-robot collaboration towards a shared goal requires robots to understand human action and interaction with the surrounding environment. This paper focuses on human-robot interaction (HRI) based on human-robot dialogue that relies on the robot action confirmation and action step generation using multimodal scene understanding. The state-of-the-art approach uses multimodal transformers to generate robot action steps aligned with robot action confirmation from a single clip showing a task composed of multiple micro steps. Although actions towards a long-horizon task depend on each other throughout an entire video, the current approaches mainly focus on clip-level processing and do not leverage long-context information. This paper proposes a long-context Q-former incorporating left and right context dependency in full videos. Furthermore, this paper proposes a text-conditioning approach to feed text embeddings directly into the LLM decoder to mitigate the high abstraction of the information in text by Q-former. Experiments with the YouCook2 corpus show that the accuracy of confirmation generation is a major factor in the performance of action planning. Furthermore, we demonstrate that the long-context Q-former improves the confirmation and action planning by integrating VideoLLaMA3. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_17335 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Robot Confirmation Generation and Action Planning Using Long-context Q-Former Integrated with Multimodal LLM Hori, Chiori Masuyama, Yoshiki Jain, Siddarth Corcodel, Radu Jha, Devesh Romeres, Diego Roux, Jonathan Le Robotics Computation and Language Computer Vision and Pattern Recognition Sound Audio and Speech Processing Human-robot collaboration towards a shared goal requires robots to understand human action and interaction with the surrounding environment. This paper focuses on human-robot interaction (HRI) based on human-robot dialogue that relies on the robot action confirmation and action step generation using multimodal scene understanding. The state-of-the-art approach uses multimodal transformers to generate robot action steps aligned with robot action confirmation from a single clip showing a task composed of multiple micro steps. Although actions towards a long-horizon task depend on each other throughout an entire video, the current approaches mainly focus on clip-level processing and do not leverage long-context information. This paper proposes a long-context Q-former incorporating left and right context dependency in full videos. Furthermore, this paper proposes a text-conditioning approach to feed text embeddings directly into the LLM decoder to mitigate the high abstraction of the information in text by Q-former. Experiments with the YouCook2 corpus show that the accuracy of confirmation generation is a major factor in the performance of action planning. Furthermore, we demonstrate that the long-context Q-former improves the confirmation and action planning by integrating VideoLLaMA3. |
| title | Robot Confirmation Generation and Action Planning Using Long-context Q-Former Integrated with Multimodal LLM |
| topic | Robotics Computation and Language Computer Vision and Pattern Recognition Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2511.17335 |