Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.23947 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908661517910016 |
|---|---|
| author | Liu, Zihan Rabbani, Parisa Duddu, Veda Fan, Kyle Lee, Madison Huang, Yun |
| author_facet | Liu, Zihan Rabbani, Parisa Duddu, Veda Fan, Kyle Lee, Madison Huang, Yun |
| contents | LLM-powered multimodal systems are increasingly used to interpret human behavior, yet how researchers apply the models' 'social competence' remains poorly understood. This paper presents a systematic literature review of 176 publications across different application domains (e.g., healthcare, education, and entertainment). Using a four-dimensional coding framework (application, technical, evaluative, and ethical), we find (1) frequent use of pattern recognition and information extraction from multimodal sources, but limited support for adaptive, interactive reasoning; (2) a dominant 'modality-to-text' pipeline that privileges language over rich audiovisual cues, striping away nuanced social cues; (3) evaluation practices reliant on static benchmarks, with socially grounded, human-centered assessments rare; and (4) Ethical discussions focused mainly on legal and rights-related risks (e.g., privacy), leaving societal risks (e.g., deception) overlooked--or at best acknowledged but left unaddressed. We outline a research agenda for evaluating socially competent, ethically informed, and interaction-aware multi-modal systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_23947 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Social Gaze of LLMs: A Literature Review of Multimodal Approaches to Human Behavior Understanding Liu, Zihan Rabbani, Parisa Duddu, Veda Fan, Kyle Lee, Madison Huang, Yun Human-Computer Interaction LLM-powered multimodal systems are increasingly used to interpret human behavior, yet how researchers apply the models' 'social competence' remains poorly understood. This paper presents a systematic literature review of 176 publications across different application domains (e.g., healthcare, education, and entertainment). Using a four-dimensional coding framework (application, technical, evaluative, and ethical), we find (1) frequent use of pattern recognition and information extraction from multimodal sources, but limited support for adaptive, interactive reasoning; (2) a dominant 'modality-to-text' pipeline that privileges language over rich audiovisual cues, striping away nuanced social cues; (3) evaluation practices reliant on static benchmarks, with socially grounded, human-centered assessments rare; and (4) Ethical discussions focused mainly on legal and rights-related risks (e.g., privacy), leaving societal risks (e.g., deception) overlooked--or at best acknowledged but left unaddressed. We outline a research agenda for evaluating socially competent, ethically informed, and interaction-aware multi-modal systems. |
| title | The Social Gaze of LLMs: A Literature Review of Multimodal Approaches to Human Behavior Understanding |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2510.23947 |