Leveraging LLMs with Iterative Loop Structure for Enhanced Social Intelligence in Video Question Answering

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Mori, Erika, Qiu, Yue, Kataoka, Hirokatsu, Aoki, Yoshimitsu
Format:	Preprint
Published:	2025
Subjects:	Computer Vision and Pattern Recognition
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866909554489425920
author	Mori, Erika Qiu, Yue Kataoka, Hirokatsu Aoki, Yoshimitsu
author_facet	Mori, Erika Qiu, Yue Kataoka, Hirokatsu Aoki, Yoshimitsu
contents	Social intelligence, the ability to interpret emotions, intentions, and behaviors, is essential for effective communication and adaptive responses. As robots and AI systems become more prevalent in caregiving, healthcare, and education, the demand for AI that can interact naturally with humans grows. However, creating AI that seamlessly integrates multiple modalities, such as vision and speech, remains a challenge. Current video-based methods for social intelligence rely on general video recognition or emotion recognition techniques, often overlook the unique elements inherent in human interactions. To address this, we propose the Looped Video Debating (LVD) framework, which integrates Large Language Models (LLMs) with visual information, such as facial expressions and body movements, to enhance the transparency and reliability of question-answering tasks involving human interaction videos. Our results on the Social-IQ 2.0 benchmark show that LVD achieves state-of-the-art performance without fine-tuning. Furthermore, supplementary human annotations on existing datasets provide insights into the model's accuracy, guiding future improvements in AI-driven social intelligence.
format	Preprint
id	arxiv_https___arxiv_org_abs_2503_21190
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	Leveraging LLMs with Iterative Loop Structure for Enhanced Social Intelligence in Video Question Answering Mori, Erika Qiu, Yue Kataoka, Hirokatsu Aoki, Yoshimitsu Computer Vision and Pattern Recognition Social intelligence, the ability to interpret emotions, intentions, and behaviors, is essential for effective communication and adaptive responses. As robots and AI systems become more prevalent in caregiving, healthcare, and education, the demand for AI that can interact naturally with humans grows. However, creating AI that seamlessly integrates multiple modalities, such as vision and speech, remains a challenge. Current video-based methods for social intelligence rely on general video recognition or emotion recognition techniques, often overlook the unique elements inherent in human interactions. To address this, we propose the Looped Video Debating (LVD) framework, which integrates Large Language Models (LLMs) with visual information, such as facial expressions and body movements, to enhance the transparency and reliability of question-answering tasks involving human interaction videos. Our results on the Social-IQ 2.0 benchmark show that LVD achieves state-of-the-art performance without fine-tuning. Furthermore, supplementary human annotations on existing datasets provide insights into the model's accuracy, guiding future improvements in AI-driven social intelligence.
title	Leveraging LLMs with Iterative Loop Structure for Enhanced Social Intelligence in Video Question Answering
topic	Computer Vision and Pattern Recognition
url	https://arxiv.org/abs/2503.21190