Social Caption: Evaluating Social Understanding in Multimodal Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915742682710016 |
|---|---|
| author | Thumu, Bhaavanaa Mathur, Leena Kebe, Youssouf Morency, Louis-Philippe |
| author_facet | Thumu, Bhaavanaa Mathur, Leena Kebe, Youssouf Morency, Louis-Philippe |
| contents | Social understanding abilities are crucial for multimodal large language models (MLLMs) to interpret human social interactions. We introduce Social Caption, a framework grounded in interaction theory to evaluate social understanding abilities of MLLMs along three dimensions: Social Inference (SI), the ability to make accurate inferences about interactions; Holistic Social Analysis (HSA), the ability to generate comprehensive descriptions of interactions; Directed Social Analysis (DSA), the ability to extract relevant social information from interactions. We analyze factors influencing model performance in social understanding, such as scale, architectural design, and spoken context. Experiments with MLLM judges contribute insights about scaling automated evaluation of multimodal social understanding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_14569 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Social Caption: Evaluating Social Understanding in Multimodal Models Thumu, Bhaavanaa Mathur, Leena Kebe, Youssouf Morency, Louis-Philippe Computation and Language Machine Learning Social understanding abilities are crucial for multimodal large language models (MLLMs) to interpret human social interactions. We introduce Social Caption, a framework grounded in interaction theory to evaluate social understanding abilities of MLLMs along three dimensions: Social Inference (SI), the ability to make accurate inferences about interactions; Holistic Social Analysis (HSA), the ability to generate comprehensive descriptions of interactions; Directed Social Analysis (DSA), the ability to extract relevant social information from interactions. We analyze factors influencing model performance in social understanding, such as scale, architectural design, and spoken context. Experiments with MLLM judges contribute insights about scaling automated evaluation of multimodal social understanding. |
| title | Social Caption: Evaluating Social Understanding in Multimodal Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2601.14569 |