VideoQA in the Era of LLMs: An Empirical Study
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913892914954240 |
|---|---|
| author | Xiao, Junbin Huang, Nanxin Qin, Hangyu Li, Dongyang Li, Yicong Zhu, Fengbin Tao, Zhulin Yu, Jianxing Lin, Liang Chua, Tat-Seng Yao, Angela |
| author_facet | Xiao, Junbin Huang, Nanxin Qin, Hangyu Li, Dongyang Li, Yicong Zhu, Fengbin Tao, Zhulin Yu, Jianxing Lin, Liang Chua, Tat-Seng Yao, Angela |
| contents | Video Large Language Models (Video-LLMs) are flourishing and has advanced many video-language tasks. As a golden testbed, Video Question Answering (VideoQA) plays pivotal role in Video-LLM developing. This work conducts a timely and comprehensive study of Video-LLMs' behavior in VideoQA, aiming to elucidate their success and failure modes, and provide insights towards more human-like video understanding and question answering. Our analyses demonstrate that Video-LLMs excel in VideoQA; they can correlate contextual cues and generate plausible responses to questions about varied video contents. However, models falter in handling video temporality, both in reasoning about temporal content ordering and grounding QA-relevant temporal moments. Moreover, the models behave unintuitively - they are unresponsive to adversarial video perturbations while being sensitive to simple variations of candidate answers and questions. Also, they do not necessarily generalize better. The findings demonstrate Video-LLMs' QA capability in standard condition yet highlight their severe deficiency in robustness and interpretability, suggesting the urgent need on rationales in Video-LLM developing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_04223 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VideoQA in the Era of LLMs: An Empirical Study Xiao, Junbin Huang, Nanxin Qin, Hangyu Li, Dongyang Li, Yicong Zhu, Fengbin Tao, Zhulin Yu, Jianxing Lin, Liang Chua, Tat-Seng Yao, Angela Computer Vision and Pattern Recognition Artificial Intelligence Video Large Language Models (Video-LLMs) are flourishing and has advanced many video-language tasks. As a golden testbed, Video Question Answering (VideoQA) plays pivotal role in Video-LLM developing. This work conducts a timely and comprehensive study of Video-LLMs' behavior in VideoQA, aiming to elucidate their success and failure modes, and provide insights towards more human-like video understanding and question answering. Our analyses demonstrate that Video-LLMs excel in VideoQA; they can correlate contextual cues and generate plausible responses to questions about varied video contents. However, models falter in handling video temporality, both in reasoning about temporal content ordering and grounding QA-relevant temporal moments. Moreover, the models behave unintuitively - they are unresponsive to adversarial video perturbations while being sensitive to simple variations of candidate answers and questions. Also, they do not necessarily generalize better. The findings demonstrate Video-LLMs' QA capability in standard condition yet highlight their severe deficiency in robustness and interpretability, suggesting the urgent need on rationales in Video-LLM developing. |
| title | VideoQA in the Era of LLMs: An Empirical Study |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2408.04223 |