Situated Instruction Following
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909259143315456 |
|---|---|
| author | Min, So Yeon Puig, Xavi Chaplot, Devendra Singh Yang, Tsung-Yen Rai, Akshara Parashar, Priyam Salakhutdinov, Ruslan Bisk, Yonatan Mottaghi, Roozbeh |
| author_facet | Min, So Yeon Puig, Xavi Chaplot, Devendra Singh Yang, Tsung-Yen Rai, Akshara Parashar, Priyam Salakhutdinov, Ruslan Bisk, Yonatan Mottaghi, Roozbeh |
| contents | Language is never spoken in a vacuum. It is expressed, comprehended, and contextualized within the holistic backdrop of the speaker's history, actions, and environment. Since humans are used to communicating efficiently with situated language, the practicality of robotic assistants hinge on their ability to understand and act upon implicit and situated instructions. In traditional instruction following paradigms, the agent acts alone in an empty house, leading to language use that is both simplified and artificially "complete." In contrast, we propose situated instruction following, which embraces the inherent underspecification and ambiguity of real-world communication with the physical presence of a human speaker. The meaning of situated instructions naturally unfold through the past actions and the expected future behaviors of the human involved. Specifically, within our settings we have instructions that (1) are ambiguously specified, (2) have temporally evolving intent, (3) can be interpreted more precisely with the agent's dynamic actions. Our experiments indicate that state-of-the-art Embodied Instruction Following (EIF) models lack holistic understanding of situated human intention. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_12061 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Situated Instruction Following Min, So Yeon Puig, Xavi Chaplot, Devendra Singh Yang, Tsung-Yen Rai, Akshara Parashar, Priyam Salakhutdinov, Ruslan Bisk, Yonatan Mottaghi, Roozbeh Human-Computer Interaction Artificial Intelligence Robotics Language is never spoken in a vacuum. It is expressed, comprehended, and contextualized within the holistic backdrop of the speaker's history, actions, and environment. Since humans are used to communicating efficiently with situated language, the practicality of robotic assistants hinge on their ability to understand and act upon implicit and situated instructions. In traditional instruction following paradigms, the agent acts alone in an empty house, leading to language use that is both simplified and artificially "complete." In contrast, we propose situated instruction following, which embraces the inherent underspecification and ambiguity of real-world communication with the physical presence of a human speaker. The meaning of situated instructions naturally unfold through the past actions and the expected future behaviors of the human involved. Specifically, within our settings we have instructions that (1) are ambiguously specified, (2) have temporally evolving intent, (3) can be interpreted more precisely with the agent's dynamic actions. Our experiments indicate that state-of-the-art Embodied Instruction Following (EIF) models lack holistic understanding of situated human intention. |
| title | Situated Instruction Following |
| topic | Human-Computer Interaction Artificial Intelligence Robotics |
| url | https://arxiv.org/abs/2407.12061 |