Look and Tell: A Dataset for Multimodal Grounding Across Egocentric and Exocentric Views
Fuente:
arXiv
Guardado en:
| Autores principales: | , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917047530684416 |
|---|---|
| author | Deichler, Anna Beskow, Jonas |
| author_facet | Deichler, Anna Beskow, Jonas |
| contents | We introduce Look and Tell, a multimodal dataset for studying referential communication across egocentric and exocentric perspectives. Using Meta Project Aria smart glasses and stationary cameras, we recorded synchronized gaze, speech, and video as 25 participants instructed a partner to identify ingredients in a kitchen. Combined with 3D scene reconstructions, this setup provides a benchmark for evaluating how different spatial representations (2D vs. 3D; ego vs. exo) affect multimodal grounding. The dataset contains 3.67 hours of recordings, including 2,707 richly annotated referential expressions, and is designed to advance the development of embodied agents that can understand and engage in situated dialogue. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_22672 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Look and Tell: A Dataset for Multimodal Grounding Across Egocentric and Exocentric Views Deichler, Anna Beskow, Jonas Computer Vision and Pattern Recognition Computation and Language Robotics I.2.10; I.2.9; I.2.7; H.5.2 We introduce Look and Tell, a multimodal dataset for studying referential communication across egocentric and exocentric perspectives. Using Meta Project Aria smart glasses and stationary cameras, we recorded synchronized gaze, speech, and video as 25 participants instructed a partner to identify ingredients in a kitchen. Combined with 3D scene reconstructions, this setup provides a benchmark for evaluating how different spatial representations (2D vs. 3D; ego vs. exo) affect multimodal grounding. The dataset contains 3.67 hours of recordings, including 2,707 richly annotated referential expressions, and is designed to advance the development of embodied agents that can understand and engage in situated dialogue. |
| title | Look and Tell: A Dataset for Multimodal Grounding Across Egocentric and Exocentric Views |
| topic | Computer Vision and Pattern Recognition Computation and Language Robotics I.2.10; I.2.9; I.2.7; H.5.2 |
| url | https://arxiv.org/abs/2510.22672 |