Exploring Audio Hallucination in Egocentric Video Understanding
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911625117696000 |
|---|---|
| author | Seth, Ashish Mei, Xinhao Zhao, Changsheng Nagaraja, Varun Chang, Ernie Meyer, Gregory P. Lan, Gael Le Xiong, Yunyang Chandra, Vikas Shi, Yangyang Manocha, Dinesh Cai, Zhipeng |
| author_facet | Seth, Ashish Mei, Xinhao Zhao, Changsheng Nagaraja, Varun Chang, Ernie Meyer, Gregory P. Lan, Gael Le Xiong, Yunyang Chandra, Vikas Shi, Yangyang Manocha, Dinesh Cai, Zhipeng |
| contents | Egocentric videos provide a distinctive setting in which sound serves as crucial cues to understand user activities and surroundings, particularly when visual information is unstable or occluded due to continuous camera movement. State-of-the-art large audio-visual language models (AV-LLMs) can generate multimodal descriptions. However, we show in this work that they are prone to audio hallucinations, often inferring sounds from visual cues that are visible but not heard. We present a systematic and automatic evaluation framework for analyzing audio hallucinations in egocentric video through a targeted question-answering (Q/A) protocol. We curate a dataset of 300 egocentric videos and design 1,000 sound-focused questions to probe model outputs. To characterize hallucinations, we propose a grounded taxonomy that distinguishes between foreground action sounds from the user activities and background ambient sounds. Our evaluation shows that advanced AV-LLMs, such as Qwen2.5 Omni, exhibit high hallucination rates, achieving only 27.3% and 39.5% accuracy on Q/As related to foreground and background sounds, respectively. With this work, we highlight the need to measure the reliability of multimodal responses, emphasizing that robust evaluation of hallucinations is essential to develop reliable AV-LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_23860 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Exploring Audio Hallucination in Egocentric Video Understanding Seth, Ashish Mei, Xinhao Zhao, Changsheng Nagaraja, Varun Chang, Ernie Meyer, Gregory P. Lan, Gael Le Xiong, Yunyang Chandra, Vikas Shi, Yangyang Manocha, Dinesh Cai, Zhipeng Computer Vision and Pattern Recognition Artificial Intelligence Egocentric videos provide a distinctive setting in which sound serves as crucial cues to understand user activities and surroundings, particularly when visual information is unstable or occluded due to continuous camera movement. State-of-the-art large audio-visual language models (AV-LLMs) can generate multimodal descriptions. However, we show in this work that they are prone to audio hallucinations, often inferring sounds from visual cues that are visible but not heard. We present a systematic and automatic evaluation framework for analyzing audio hallucinations in egocentric video through a targeted question-answering (Q/A) protocol. We curate a dataset of 300 egocentric videos and design 1,000 sound-focused questions to probe model outputs. To characterize hallucinations, we propose a grounded taxonomy that distinguishes between foreground action sounds from the user activities and background ambient sounds. Our evaluation shows that advanced AV-LLMs, such as Qwen2.5 Omni, exhibit high hallucination rates, achieving only 27.3% and 39.5% accuracy on Q/As related to foreground and background sounds, respectively. With this work, we highlight the need to measure the reliability of multimodal responses, emphasizing that robust evaluation of hallucinations is essential to develop reliable AV-LLMs. |
| title | Exploring Audio Hallucination in Egocentric Video Understanding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2604.23860 |