VideoAgent: Long-form Video Understanding with Large Language Model as Agent
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929278326669312 |
|---|---|
| author | Wang, Xiaohan Zhang, Yuhui Zohar, Orr Yeung-Levy, Serena |
| author_facet | Wang, Xiaohan Zhang, Yuhui Zohar, Orr Yeung-Levy, Serena |
| contents | Long-form video understanding represents a significant challenge within computer vision, demanding a model capable of reasoning over long multi-modal sequences. Motivated by the human cognitive process for long-form video understanding, we emphasize interactive reasoning and planning over the ability to process lengthy visual inputs. We introduce a novel agent-based system, VideoAgent, that employs a large language model as a central agent to iteratively identify and compile crucial information to answer a question, with vision-language foundation models serving as tools to translate and retrieve visual information. Evaluated on the challenging EgoSchema and NExT-QA benchmarks, VideoAgent achieves 54.1% and 71.3% zero-shot accuracy with only 8.4 and 8.2 frames used on average. These results demonstrate superior effectiveness and efficiency of our method over the current state-of-the-art methods, highlighting the potential of agent-based approaches in advancing long-form video understanding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_10517 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VideoAgent: Long-form Video Understanding with Large Language Model as Agent Wang, Xiaohan Zhang, Yuhui Zohar, Orr Yeung-Levy, Serena Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Information Retrieval Long-form video understanding represents a significant challenge within computer vision, demanding a model capable of reasoning over long multi-modal sequences. Motivated by the human cognitive process for long-form video understanding, we emphasize interactive reasoning and planning over the ability to process lengthy visual inputs. We introduce a novel agent-based system, VideoAgent, that employs a large language model as a central agent to iteratively identify and compile crucial information to answer a question, with vision-language foundation models serving as tools to translate and retrieve visual information. Evaluated on the challenging EgoSchema and NExT-QA benchmarks, VideoAgent achieves 54.1% and 71.3% zero-shot accuracy with only 8.4 and 8.2 frames used on average. These results demonstrate superior effectiveness and efficiency of our method over the current state-of-the-art methods, highlighting the potential of agent-based approaches in advancing long-form video understanding. |
| title | VideoAgent: Long-form Video Understanding with Large Language Model as Agent |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2403.10517 |