GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911688154939392 |
|---|---|
| author | Kim, Junho Cao, Xu Yang, Houze Boote, Bikram Jojic, Ana Ryan, Fiona Lai, Bolin Lee, Sangmin Rehg, James M. |
| author_facet | Kim, Junho Cao, Xu Yang, Houze Boote, Bikram Jojic, Ana Ryan, Fiona Lai, Bolin Lee, Sangmin Rehg, James M. |
| contents | Understanding social interactions requires reasoning over subtle non-verbal cues, yet current multimodal large language models (MLLMs) often fail to identify who interacts with whom in multi-person videos. We introduce GRASP, a large-scale social reasoning dataset that connects high-level social QA with fine-grained gaze and deictic gesture events. GRASP contains 290K question--answer pairs over 46K videos totaling 749 hours, organized by a 16-category taxonomy spanning gaze, gesture, and joint gaze--gesture reasoning, together with GRASP-Bench for evaluation. Unlike prior resources that focus on either isolated cues or high-level social QA, GRASP builds questions from identity-consistent gaze trajectories, deictic gestures, and their joint compositions into social events. Moreover, we propose Social Grounding Reward (SGR), a learning signal that uses these social events to encourage models to reason about the participants involved in each interaction. Experiments show that SGR improves performance on GRASP-Bench while maintaining zero-shot performance on related social video QA benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_15764 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions Kim, Junho Cao, Xu Yang, Houze Boote, Bikram Jojic, Ana Ryan, Fiona Lai, Bolin Lee, Sangmin Rehg, James M. Computer Vision and Pattern Recognition Artificial Intelligence Understanding social interactions requires reasoning over subtle non-verbal cues, yet current multimodal large language models (MLLMs) often fail to identify who interacts with whom in multi-person videos. We introduce GRASP, a large-scale social reasoning dataset that connects high-level social QA with fine-grained gaze and deictic gesture events. GRASP contains 290K question--answer pairs over 46K videos totaling 749 hours, organized by a 16-category taxonomy spanning gaze, gesture, and joint gaze--gesture reasoning, together with GRASP-Bench for evaluation. Unlike prior resources that focus on either isolated cues or high-level social QA, GRASP builds questions from identity-consistent gaze trajectories, deictic gestures, and their joint compositions into social events. Moreover, we propose Social Grounding Reward (SGR), a learning signal that uses these social events to encourage models to reason about the participants involved in each interaction. Experiments show that SGR improves performance on GRASP-Bench while maintaining zero-shot performance on related social video QA benchmarks. |
| title | GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2605.15764 |