MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908316069789696 |
|---|---|
| author | Ye, Hanrong Zhang, Haotian Daxberger, Erik Chen, Lin Lin, Zongyu Li, Yanghao Zhang, Bowen You, Haoxuan Xu, Dan Gan, Zhe Lu, Jiasen Yang, Yinfei |
| author_facet | Ye, Hanrong Zhang, Haotian Daxberger, Erik Chen, Lin Lin, Zongyu Li, Yanghao Zhang, Bowen You, Haoxuan Xu, Dan Gan, Zhe Lu, Jiasen Yang, Yinfei |
| contents | This research aims to comprehensively explore building a multimodal foundation model for egocentric video understanding. To achieve this goal, we work on three fronts. First, as there is a lack of QA data for egocentric video understanding, we automatically generate 7M high-quality QA samples for egocentric videos ranging from 30 seconds to one hour long in Ego4D based on human-annotated data. This is one of the largest egocentric QA datasets. Second, we contribute a challenging egocentric QA benchmark with 629 videos and 7,026 questions to evaluate the models' ability in recognizing and memorizing visual details across videos of varying lengths. We introduce a new de-biasing evaluation method to help mitigate the unavoidable language bias present in the models being evaluated. Third, we propose a specialized multimodal architecture featuring a novel "Memory Pointer Prompting" mechanism. This design includes a \textit{global glimpse} step to gain an overarching understanding of the entire video and identify key visual information, followed by a fallback step that utilizes the key visual information to generate responses. This enables the model to more effectively comprehend extended video content. With the data, benchmark, and model, we build MM-Ego, an egocentric multimodal LLM that shows powerful performance on egocentric video understanding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07177 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA Ye, Hanrong Zhang, Haotian Daxberger, Erik Chen, Lin Lin, Zongyu Li, Yanghao Zhang, Bowen You, Haoxuan Xu, Dan Gan, Zhe Lu, Jiasen Yang, Yinfei Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning This research aims to comprehensively explore building a multimodal foundation model for egocentric video understanding. To achieve this goal, we work on three fronts. First, as there is a lack of QA data for egocentric video understanding, we automatically generate 7M high-quality QA samples for egocentric videos ranging from 30 seconds to one hour long in Ego4D based on human-annotated data. This is one of the largest egocentric QA datasets. Second, we contribute a challenging egocentric QA benchmark with 629 videos and 7,026 questions to evaluate the models' ability in recognizing and memorizing visual details across videos of varying lengths. We introduce a new de-biasing evaluation method to help mitigate the unavoidable language bias present in the models being evaluated. Third, we propose a specialized multimodal architecture featuring a novel "Memory Pointer Prompting" mechanism. This design includes a \textit{global glimpse} step to gain an overarching understanding of the entire video and identify key visual information, followed by a fallback step that utilizes the key visual information to generate responses. This enables the model to more effectively comprehend extended video content. With the data, benchmark, and model, we build MM-Ego, an egocentric multimodal LLM that shows powerful performance on egocentric video understanding. |
| title | MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2410.07177 |