Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.10525 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910606592835584 |
|---|---|
| author | Bohus, Dan Andrist, Sean Bao, Yuwei Horvitz, Eric Paradiso, Ann |
| author_facet | Bohus, Dan Andrist, Sean Bao, Yuwei Horvitz, Eric Paradiso, Ann |
| contents | We report initial work towards constructing ecologically valid benchmarks to assess the capabilities of large multimodal models for engaging in situated collaboration. In contrast to existing benchmarks, in which question-answer pairs are generated post hoc over preexisting or synthetic datasets via templates, human annotators, or large language models (LLMs), we propose and investigate an interactive system-driven approach, where the questions are generated by users in context, during their interactions with an end-to-end situated AI system. We illustrate how the questions that arise are different in form and content from questions typically found in existing embodied question answering (EQA) benchmarks and discuss new real-world challenge problems brought to the fore. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_10525 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | "Is This It?": Towards Ecologically Valid Benchmarks for Situated Collaboration Bohus, Dan Andrist, Sean Bao, Yuwei Horvitz, Eric Paradiso, Ann Multimedia Artificial Intelligence Computation and Language We report initial work towards constructing ecologically valid benchmarks to assess the capabilities of large multimodal models for engaging in situated collaboration. In contrast to existing benchmarks, in which question-answer pairs are generated post hoc over preexisting or synthetic datasets via templates, human annotators, or large language models (LLMs), we propose and investigate an interactive system-driven approach, where the questions are generated by users in context, during their interactions with an end-to-end situated AI system. We illustrate how the questions that arise are different in form and content from questions typically found in existing embodied question answering (EQA) benchmarks and discuss new real-world challenge problems brought to the fore. |
| title | "Is This It?": Towards Ecologically Valid Benchmarks for Situated Collaboration |
| topic | Multimedia Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2409.10525 |