Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.10360 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910744000331776 |
|---|---|
| author | Zohar, Orr Wang, Xiaohan Dubois, Yann Mehta, Nikhil Xiao, Tong Hansen-Estruch, Philippe Yu, Licheng Wang, Xiaofang Juefei-Xu, Felix Zhang, Ning Yeung-Levy, Serena Xia, Xide |
| author_facet | Zohar, Orr Wang, Xiaohan Dubois, Yann Mehta, Nikhil Xiao, Tong Hansen-Estruch, Philippe Yu, Licheng Wang, Xiaofang Juefei-Xu, Felix Zhang, Ning Yeung-Levy, Serena Xia, Xide |
| contents | Despite the rapid integration of video perception capabilities into Large Multimodal Models (LMMs), the underlying mechanisms driving their video understanding remain poorly understood. Consequently, many design decisions in this domain are made without proper justification or analysis. The high computational cost of training and evaluating such models, coupled with limited open research, hinders the development of video-LMMs. To address this, we present a comprehensive study that helps uncover what effectively drives video understanding in LMMs.
We begin by critically examining the primary contributors to the high computational requirements associated with video-LMM research and discover Scaling Consistency, wherein design and training decisions made on smaller models and datasets (up to a critical size) effectively transfer to larger models. Leveraging these insights, we explored many video-specific aspects of video-LMMs, including video sampling, architectures, data composition, training schedules, and more. For example, we demonstrated that fps sampling during training is vastly preferable to uniform frame sampling and which vision encoders are the best for video representation.
Guided by these findings, we introduce Apollo, a state-of-the-art family of LMMs that achieve superior performance across different model sizes. Our models can perceive hour-long videos efficiently, with Apollo-3B outperforming most existing $7$B models with an impressive 55.1 on LongVideoBench. Apollo-7B is state-of-the-art compared to 7B LMMs with a 70.9 on MLVU, and 63.3 on Video-MME. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_10360 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Apollo: An Exploration of Video Understanding in Large Multimodal Models Zohar, Orr Wang, Xiaohan Dubois, Yann Mehta, Nikhil Xiao, Tong Hansen-Estruch, Philippe Yu, Licheng Wang, Xiaofang Juefei-Xu, Felix Zhang, Ning Yeung-Levy, Serena Xia, Xide Computer Vision and Pattern Recognition Artificial Intelligence Despite the rapid integration of video perception capabilities into Large Multimodal Models (LMMs), the underlying mechanisms driving their video understanding remain poorly understood. Consequently, many design decisions in this domain are made without proper justification or analysis. The high computational cost of training and evaluating such models, coupled with limited open research, hinders the development of video-LMMs. To address this, we present a comprehensive study that helps uncover what effectively drives video understanding in LMMs. We begin by critically examining the primary contributors to the high computational requirements associated with video-LMM research and discover Scaling Consistency, wherein design and training decisions made on smaller models and datasets (up to a critical size) effectively transfer to larger models. Leveraging these insights, we explored many video-specific aspects of video-LMMs, including video sampling, architectures, data composition, training schedules, and more. For example, we demonstrated that fps sampling during training is vastly preferable to uniform frame sampling and which vision encoders are the best for video representation. Guided by these findings, we introduce Apollo, a state-of-the-art family of LMMs that achieve superior performance across different model sizes. Our models can perceive hour-long videos efficiently, with Apollo-3B outperforming most existing $7$B models with an impressive 55.1 on LongVideoBench. Apollo-7B is state-of-the-art compared to 7B LMMs with a 70.9 on MLVU, and 63.3 on Video-MME. |
| title | Apollo: An Exploration of Video Understanding in Large Multimodal Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2412.10360 |