EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910897723670528 |
|---|---|
| author | Li, Yuxuan Veerabadran, Vijay Iuzzolino, Michael L. Roads, Brett D. Celikyilmaz, Asli Ridgeway, Karl |
| author_facet | Li, Yuxuan Veerabadran, Vijay Iuzzolino, Michael L. Roads, Brett D. Celikyilmaz, Asli Ridgeway, Karl |
| contents | We introduce EgoToM, a new video question-answering benchmark that extends Theory-of-Mind (ToM) evaluation to egocentric domains. Using a causal ToM model, we generate multi-choice video QA instances for the Ego4D dataset to benchmark the ability to predict a camera wearer's goals, beliefs, and next actions. We study the performance of both humans and state of the art multimodal large language models (MLLMs) on these three interconnected inference problems. Our evaluation shows that MLLMs achieve close to human-level accuracy on inferring goals from egocentric videos. However, MLLMs (including the largest ones we tested with over 100B parameters) fall short of human performance when inferring the camera wearers' in-the-moment belief states and future actions that are most consistent with the unseen video future. We believe that our results will shape the future design of an important class of egocentric digital assistants which are equipped with a reasonable model of the user's internal mental states. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_22152 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos Li, Yuxuan Veerabadran, Vijay Iuzzolino, Michael L. Roads, Brett D. Celikyilmaz, Asli Ridgeway, Karl Computer Vision and Pattern Recognition Artificial Intelligence We introduce EgoToM, a new video question-answering benchmark that extends Theory-of-Mind (ToM) evaluation to egocentric domains. Using a causal ToM model, we generate multi-choice video QA instances for the Ego4D dataset to benchmark the ability to predict a camera wearer's goals, beliefs, and next actions. We study the performance of both humans and state of the art multimodal large language models (MLLMs) on these three interconnected inference problems. Our evaluation shows that MLLMs achieve close to human-level accuracy on inferring goals from egocentric videos. However, MLLMs (including the largest ones we tested with over 100B parameters) fall short of human performance when inferring the camera wearers' in-the-moment belief states and future actions that are most consistent with the unseen video future. We believe that our results will shape the future design of an important class of egocentric digital assistants which are equipped with a reasonable model of the user's internal mental states. |
| title | EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2503.22152 |