GEXIA: Granularity Expansion and Iterative Approximation for Scalable Multi-grained Video-language Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916517078106112 |
|---|---|
| author | Wang, Yicheng Zhang, Zhikang Wang, Jue Fan, David Xu, Zhenlin Liu, Linda Hao, Xiang Bhat, Vimal Li, Xinyu |
| author_facet | Wang, Yicheng Zhang, Zhikang Wang, Jue Fan, David Xu, Zhenlin Liu, Linda Hao, Xiang Bhat, Vimal Li, Xinyu |
| contents | In various video-language learning tasks, the challenge of achieving cross-modality alignment with multi-grained data persists. We propose a method to tackle this challenge from two crucial perspectives: data and modeling. Given the absence of a multi-grained video-text pretraining dataset, we introduce a Granularity EXpansion (GEX) method with Integration and Compression operations to expand the granularity of a single-grained dataset. To better model multi-grained data, we introduce an Iterative Approximation Module (IAM), which embeds multi-grained videos and texts into a unified, low-dimensional semantic space while preserving essential information for cross-modal alignment. Furthermore, GEXIA is highly scalable with no restrictions on the number of video-text granularities for alignment. We evaluate our work on three categories of video tasks across seven benchmark datasets, showcasing state-of-the-art or comparable performance. Remarkably, our model excels in tasks involving long-form video understanding, even though the pretraining dataset only contains short video clips. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_07704 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GEXIA: Granularity Expansion and Iterative Approximation for Scalable Multi-grained Video-language Learning Wang, Yicheng Zhang, Zhikang Wang, Jue Fan, David Xu, Zhenlin Liu, Linda Hao, Xiang Bhat, Vimal Li, Xinyu Computer Vision and Pattern Recognition In various video-language learning tasks, the challenge of achieving cross-modality alignment with multi-grained data persists. We propose a method to tackle this challenge from two crucial perspectives: data and modeling. Given the absence of a multi-grained video-text pretraining dataset, we introduce a Granularity EXpansion (GEX) method with Integration and Compression operations to expand the granularity of a single-grained dataset. To better model multi-grained data, we introduce an Iterative Approximation Module (IAM), which embeds multi-grained videos and texts into a unified, low-dimensional semantic space while preserving essential information for cross-modal alignment. Furthermore, GEXIA is highly scalable with no restrictions on the number of video-text granularities for alignment. We evaluate our work on three categories of video tasks across seven benchmark datasets, showcasing state-of-the-art or comparable performance. Remarkably, our model excels in tasks involving long-form video understanding, even though the pretraining dataset only contains short video clips. |
| title | GEXIA: Granularity Expansion and Iterative Approximation for Scalable Multi-grained Video-language Learning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.07704 |