Omni-MMSI: Toward Identity-attributed Social Interaction Understanding

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Xinpeng, Lai, Bolin, Chen, Hardy, Deng, Shijian, Xie, Cihang, Zhou, Yuyin, Rehg, James Matthew, Tian, Yapeng
Format:	Preprint
Published:	2026
Subjects:	Computer Vision and Pattern Recognition
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866915904277708800
author	Li, Xinpeng Lai, Bolin Chen, Hardy Deng, Shijian Xie, Cihang Zhou, Yuyin Rehg, James Matthew Tian, Yapeng
author_facet	Li, Xinpeng Lai, Bolin Chen, Hardy Deng, Shijian Xie, Cihang Zhou, Yuyin Rehg, James Matthew Tian, Yapeng
contents	We introduce Omni-MMSI, a new task that requires comprehensive social interaction understanding from raw audio, vision, and speech input. The task involves perceiving identity-attributed social cues (e.g., who is speaking what) and reasoning about the social interaction (e.g., whom the speaker refers to). This task is essential for developing AI assistants that can perceive and respond to human interactions. Unlike prior studies that operate on oracle-preprocessed social cues, Omni-MMSI reflects realistic scenarios where AI assistants must perceive and reason from raw data. However, existing pipelines and multi-modal LLMs perform poorly on Omni-MMSI because they lack reliable identity attribution capabilities, which leads to inaccurate social interaction understanding. To address this challenge, we propose Omni-MMSI-R, a reference-guided pipeline that produces identity-attributed social cues with tools and conducts chain-of-thought social reasoning. To facilitate this pipeline, we construct participant-level reference pairs and curate reasoning annotations on top of the existing datasets. Experiments demonstrate that Omni-MMSI-R outperforms advanced LLMs and counterparts on Omni-MMSI. Project page: https://sampson-lee.github.io/omni-mmsi-project-page.
format	Preprint
id	arxiv_https___arxiv_org_abs_2604_00267
institution	arXiv
publishDate	2026
record_format	arxiv
spellingShingle	Omni-MMSI: Toward Identity-attributed Social Interaction Understanding Li, Xinpeng Lai, Bolin Chen, Hardy Deng, Shijian Xie, Cihang Zhou, Yuyin Rehg, James Matthew Tian, Yapeng Computer Vision and Pattern Recognition We introduce Omni-MMSI, a new task that requires comprehensive social interaction understanding from raw audio, vision, and speech input. The task involves perceiving identity-attributed social cues (e.g., who is speaking what) and reasoning about the social interaction (e.g., whom the speaker refers to). This task is essential for developing AI assistants that can perceive and respond to human interactions. Unlike prior studies that operate on oracle-preprocessed social cues, Omni-MMSI reflects realistic scenarios where AI assistants must perceive and reason from raw data. However, existing pipelines and multi-modal LLMs perform poorly on Omni-MMSI because they lack reliable identity attribution capabilities, which leads to inaccurate social interaction understanding. To address this challenge, we propose Omni-MMSI-R, a reference-guided pipeline that produces identity-attributed social cues with tools and conducts chain-of-thought social reasoning. To facilitate this pipeline, we construct participant-level reference pairs and curate reasoning annotations on top of the existing datasets. Experiments demonstrate that Omni-MMSI-R outperforms advanced LLMs and counterparts on Omni-MMSI. Project page: https://sampson-lee.github.io/omni-mmsi-project-page.
title	Omni-MMSI: Toward Identity-attributed Social Interaction Understanding
topic	Computer Vision and Pattern Recognition
url	https://arxiv.org/abs/2604.00267