Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.11318 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911999890292736 |
|---|---|
| author | Lee, Hyeongmin Kim, Jin-Young Baek, Kyungjune Kim, Jihwan Go, Hyojun Ha, Seongsu Han, Seokjin Jang, Jiho Jung, Raehyuk Kim, Daewoo Kim, GeunOh Kim, JongMok Kim, Jongseok Kim, Junwan Kwon, Soonwoo Lee, Jangwon Park, Seungjoon Seo, Minjoon Suh, Jay Yi, Jaehyuk Lee, Aiden |
| author_facet | Lee, Hyeongmin Kim, Jin-Young Baek, Kyungjune Kim, Jihwan Go, Hyojun Ha, Seongsu Han, Seokjin Jang, Jiho Jung, Raehyuk Kim, Daewoo Kim, GeunOh Kim, JongMok Kim, Jongseok Kim, Junwan Kwon, Soonwoo Lee, Jangwon Park, Seungjoon Seo, Minjoon Suh, Jay Yi, Jaehyuk Lee, Aiden |
| contents | In this work, we discuss evaluating video foundation models in a fair and robust manner. Unlike language or image foundation models, many video foundation models are evaluated with differing parameters (such as sampling rate, number of frames, pretraining steps, etc.), making fair and robust comparisons challenging. Therefore, we present a carefully designed evaluation framework for measuring two core capabilities of video comprehension: appearance and motion understanding. Our findings reveal that existing video foundation models, whether text-supervised like UMT or InternVideo2, or self-supervised like V-JEPA, exhibit limitations in at least one of these capabilities. As an alternative, we introduce TWLV-I, a new video foundation model that constructs robust visual representations for both motion- and appearance-based videos. Based on the average top-1 accuracy of linear probing on five action recognition benchmarks, pretrained only on publicly accessible datasets, our model shows a 4.6%p improvement compared to V-JEPA (ViT-L) and a 7.7%p improvement compared to UMT (ViT-L). Even when compared to much larger models, our model demonstrates a 7.2%p improvement compared to DFN (ViT-H), a 2.7%p improvement compared to V-JEPA (ViT-H) and a 2.8%p improvement compared to InternVideo2 (ViT-g). We provide embedding vectors obtained by TWLV-I from videos of several commonly used video benchmarks, along with evaluation source code that can directly utilize these embeddings. The code is available at https://github.com/twelvelabs-io/video-embeddings-evaluation-framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_11318 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TWLV-I: Analysis and Insights from Holistic Evaluation on Video Foundation Models Lee, Hyeongmin Kim, Jin-Young Baek, Kyungjune Kim, Jihwan Go, Hyojun Ha, Seongsu Han, Seokjin Jang, Jiho Jung, Raehyuk Kim, Daewoo Kim, GeunOh Kim, JongMok Kim, Jongseok Kim, Junwan Kwon, Soonwoo Lee, Jangwon Park, Seungjoon Seo, Minjoon Suh, Jay Yi, Jaehyuk Lee, Aiden Computer Vision and Pattern Recognition In this work, we discuss evaluating video foundation models in a fair and robust manner. Unlike language or image foundation models, many video foundation models are evaluated with differing parameters (such as sampling rate, number of frames, pretraining steps, etc.), making fair and robust comparisons challenging. Therefore, we present a carefully designed evaluation framework for measuring two core capabilities of video comprehension: appearance and motion understanding. Our findings reveal that existing video foundation models, whether text-supervised like UMT or InternVideo2, or self-supervised like V-JEPA, exhibit limitations in at least one of these capabilities. As an alternative, we introduce TWLV-I, a new video foundation model that constructs robust visual representations for both motion- and appearance-based videos. Based on the average top-1 accuracy of linear probing on five action recognition benchmarks, pretrained only on publicly accessible datasets, our model shows a 4.6%p improvement compared to V-JEPA (ViT-L) and a 7.7%p improvement compared to UMT (ViT-L). Even when compared to much larger models, our model demonstrates a 7.2%p improvement compared to DFN (ViT-H), a 2.7%p improvement compared to V-JEPA (ViT-H) and a 2.8%p improvement compared to InternVideo2 (ViT-g). We provide embedding vectors obtained by TWLV-I from videos of several commonly used video benchmarks, along with evaluation source code that can directly utilize these embeddings. The code is available at https://github.com/twelvelabs-io/video-embeddings-evaluation-framework. |
| title | TWLV-I: Analysis and Insights from Holistic Evaluation on Video Foundation Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.11318 |