EA-VTR: Event-Aware Video-Text Retrieval
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914864857874432 |
|---|---|
| author | Ma, Zongyang Zhang, Ziqi Chen, Yuxin Qi, Zhongang Yuan, Chunfeng Li, Bing Luo, Yingmin Li, Xu Qi, Xiaojuan Shan, Ying Hu, Weiming |
| author_facet | Ma, Zongyang Zhang, Ziqi Chen, Yuxin Qi, Zhongang Yuan, Chunfeng Li, Bing Luo, Yingmin Li, Xu Qi, Xiaojuan Shan, Ying Hu, Weiming |
| contents | Understanding the content of events occurring in the video and their inherent temporal logic is crucial for video-text retrieval. However, web-crawled pre-training datasets often lack sufficient event information, and the widely adopted video-level cross-modal contrastive learning also struggles to capture detailed and complex video-text event alignment. To address these challenges, we make improvements from both data and model perspectives. In terms of pre-training data, we focus on supplementing the missing specific event content and event temporal transitions with the proposed event augmentation strategies. Based on the event-augmented data, we construct a novel Event-Aware Video-Text Retrieval model, ie, EA-VTR, which achieves powerful video-text retrieval ability through superior video event awareness. EA-VTR can efficiently encode frame-level and video-level visual representations simultaneously, enabling detailed event content and complex event temporal cross-modal alignment, ultimately enhancing the comprehensive understanding of video events. Our method not only significantly outperforms existing approaches on multiple datasets for Text-to-Video Retrieval and Video Action Recognition tasks, but also demonstrates superior event content perceive ability on Multi-event Video-Text Retrieval and Video Moment Retrieval tasks, as well as outstanding event temporal logic understanding ability on Test of Time task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_07478 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | EA-VTR: Event-Aware Video-Text Retrieval Ma, Zongyang Zhang, Ziqi Chen, Yuxin Qi, Zhongang Yuan, Chunfeng Li, Bing Luo, Yingmin Li, Xu Qi, Xiaojuan Shan, Ying Hu, Weiming Computer Vision and Pattern Recognition Understanding the content of events occurring in the video and their inherent temporal logic is crucial for video-text retrieval. However, web-crawled pre-training datasets often lack sufficient event information, and the widely adopted video-level cross-modal contrastive learning also struggles to capture detailed and complex video-text event alignment. To address these challenges, we make improvements from both data and model perspectives. In terms of pre-training data, we focus on supplementing the missing specific event content and event temporal transitions with the proposed event augmentation strategies. Based on the event-augmented data, we construct a novel Event-Aware Video-Text Retrieval model, ie, EA-VTR, which achieves powerful video-text retrieval ability through superior video event awareness. EA-VTR can efficiently encode frame-level and video-level visual representations simultaneously, enabling detailed event content and complex event temporal cross-modal alignment, ultimately enhancing the comprehensive understanding of video events. Our method not only significantly outperforms existing approaches on multiple datasets for Text-to-Video Retrieval and Video Action Recognition tasks, but also demonstrates superior event content perceive ability on Multi-event Video-Text Retrieval and Video Moment Retrieval tasks, as well as outstanding event temporal logic understanding ability on Test of Time task. |
| title | EA-VTR: Event-Aware Video-Text Retrieval |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.07478 |