Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agents
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917042739740672 |
|---|---|
| author | Veerabadran, Vijay Xiao, Fanyi Kamra, Nitin Matias, Pedro Chen, Joy Drooff, Caley Roads, Brett D Williams, Riley Henderson, Ethan Zhao, Xuanyi Carlberg, Kevin Tighe, Joseph Ridgeway, Karl |
| author_facet | Veerabadran, Vijay Xiao, Fanyi Kamra, Nitin Matias, Pedro Chen, Joy Drooff, Caley Roads, Brett D Williams, Riley Henderson, Ethan Zhao, Xuanyi Carlberg, Kevin Tighe, Joseph Ridgeway, Karl |
| contents | There has been a surge of interest in assistive wearable agents: agents embodied in wearable form factors (e.g., smart glasses) who take assistive actions toward a user's goal/query (e.g. "Where did I leave my keys?"). In this work, we consider the important complementary problem of inferring that goal from multi-modal contextual observations. Solving this "goal inference" problem holds the promise of eliminating the effort needed to interact with such an agent. This work focuses on creating WAGIBench, a strong benchmark to measure progress in solving this problem using vision-language models (VLMs). Given the limited prior work in this area, we collected a novel dataset comprising 29 hours of multimodal data from 348 participants across 3,477 recordings, featuring ground-truth goals alongside accompanying visual, audio, digital, and longitudinal contextual observations. We validate that human performance exceeds model performance, achieving 93% multiple-choice accuracy compared with 84% for the best-performing VLM. Generative benchmark results that evaluate several families of modern vision-language models show that larger models perform significantly better on the task, yet remain far from practical usefulness, as they produce relevant goals only 55% of the time. Through a modality ablation, we show that models benefit from extra information in relevant modalities with minimal performance degradation from irrelevant modalities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_22443 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agents Veerabadran, Vijay Xiao, Fanyi Kamra, Nitin Matias, Pedro Chen, Joy Drooff, Caley Roads, Brett D Williams, Riley Henderson, Ethan Zhao, Xuanyi Carlberg, Kevin Tighe, Joseph Ridgeway, Karl Computer Vision and Pattern Recognition Machine Learning There has been a surge of interest in assistive wearable agents: agents embodied in wearable form factors (e.g., smart glasses) who take assistive actions toward a user's goal/query (e.g. "Where did I leave my keys?"). In this work, we consider the important complementary problem of inferring that goal from multi-modal contextual observations. Solving this "goal inference" problem holds the promise of eliminating the effort needed to interact with such an agent. This work focuses on creating WAGIBench, a strong benchmark to measure progress in solving this problem using vision-language models (VLMs). Given the limited prior work in this area, we collected a novel dataset comprising 29 hours of multimodal data from 348 participants across 3,477 recordings, featuring ground-truth goals alongside accompanying visual, audio, digital, and longitudinal contextual observations. We validate that human performance exceeds model performance, achieving 93% multiple-choice accuracy compared with 84% for the best-performing VLM. Generative benchmark results that evaluate several families of modern vision-language models show that larger models perform significantly better on the task, yet remain far from practical usefulness, as they produce relevant goals only 55% of the time. Through a modality ablation, we show that models benefit from extra information in relevant modalities with minimal performance degradation from irrelevant modalities. |
| title | Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agents |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2510.22443 |