VideoPrism: A Foundational Visual Encoder for Video Understanding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918047777816576 |
|---|---|
| author | Zhao, Long Gundavarapu, Nitesh B. Yuan, Liangzhe Zhou, Hao Yan, Shen Sun, Jennifer J. Friedman, Luke Qian, Rui Weyand, Tobias Zhao, Yue Hornung, Rachel Schroff, Florian Yang, Ming-Hsuan Ross, David A. Wang, Huisheng Adam, Hartwig Sirotenko, Mikhail Liu, Ting Gong, Boqing |
| author_facet | Zhao, Long Gundavarapu, Nitesh B. Yuan, Liangzhe Zhou, Hao Yan, Shen Sun, Jennifer J. Friedman, Luke Qian, Rui Weyand, Tobias Zhao, Yue Hornung, Rachel Schroff, Florian Yang, Ming-Hsuan Ross, David A. Wang, Huisheng Adam, Hartwig Sirotenko, Mikhail Liu, Ting Gong, Boqing |
| contents | We introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. We pretrain VideoPrism on a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M video clips with noisy parallel text (e.g., ASR transcripts). The pretraining approach improves upon masked autoencoding by global-local distillation of semantic video embeddings and a token shuffling scheme, enabling VideoPrism to focus primarily on the video modality while leveraging the invaluable text associated with videos. We extensively test VideoPrism on four broad groups of video understanding tasks, from web video question answering to CV for science, achieving state-of-the-art performance on 31 out of 33 video understanding benchmarks. Our models are released at https://github.com/google-deepmind/videoprism. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_13217 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VideoPrism: A Foundational Visual Encoder for Video Understanding Zhao, Long Gundavarapu, Nitesh B. Yuan, Liangzhe Zhou, Hao Yan, Shen Sun, Jennifer J. Friedman, Luke Qian, Rui Weyand, Tobias Zhao, Yue Hornung, Rachel Schroff, Florian Yang, Ming-Hsuan Ross, David A. Wang, Huisheng Adam, Hartwig Sirotenko, Mikhail Liu, Ting Gong, Boqing Computer Vision and Pattern Recognition Artificial Intelligence We introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. We pretrain VideoPrism on a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M video clips with noisy parallel text (e.g., ASR transcripts). The pretraining approach improves upon masked autoencoding by global-local distillation of semantic video embeddings and a token shuffling scheme, enabling VideoPrism to focus primarily on the video modality while leveraging the invaluable text associated with videos. We extensively test VideoPrism on four broad groups of video understanding tasks, from web video question answering to CV for science, achieving state-of-the-art performance on 31 out of 33 video understanding benchmarks. Our models are released at https://github.com/google-deepmind/videoprism. |
| title | VideoPrism: A Foundational Visual Encoder for Video Understanding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2402.13217 |