SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914661012602880 |
|---|---|
| author | Dong, Xingning Guo, Qingpei Gan, Tian Wang, Qing Wu, Jianlong Ren, Xiangyuan Cheng, Yuan Chu, Wei |
| author_facet | Dong, Xingning Guo, Qingpei Gan, Tian Wang, Qing Wu, Jianlong Ren, Xiangyuan Cheng, Yuan Chu, Wei |
| contents | We present a framework for learning cross-modal video representations by directly pre-training on raw data to facilitate various downstream video-text tasks. Our main contributions lie in the pre-training framework and proxy tasks. First, based on the shortcomings of two mainstream pixel-level pre-training architectures (limited applications or less efficient), we propose Shared Network Pre-training (SNP). By employing one shared BERT-type network to refine textual and cross-modal features simultaneously, SNP is lightweight and could support various downstream applications. Second, based on the intuition that people always pay attention to several "significant words" when understanding a sentence, we propose the Significant Semantic Strengthening (S3) strategy, which includes a novel masking and matching proxy task to promote the pre-training performance. Experiments conducted on three downstream video-text tasks and six datasets demonstrate that, we establish a new state-of-the-art in pixel-level video-text pre-training; we also achieve a satisfactory balance between the pre-training efficiency and the fine-tuning performance. The codebase are available at https://github.com/alipay/Ant-Multi-Modal-Framework/tree/main/prj/snps3_vtp. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_17773 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks Dong, Xingning Guo, Qingpei Gan, Tian Wang, Qing Wu, Jianlong Ren, Xiangyuan Cheng, Yuan Chu, Wei Computer Vision and Pattern Recognition Multimedia We present a framework for learning cross-modal video representations by directly pre-training on raw data to facilitate various downstream video-text tasks. Our main contributions lie in the pre-training framework and proxy tasks. First, based on the shortcomings of two mainstream pixel-level pre-training architectures (limited applications or less efficient), we propose Shared Network Pre-training (SNP). By employing one shared BERT-type network to refine textual and cross-modal features simultaneously, SNP is lightweight and could support various downstream applications. Second, based on the intuition that people always pay attention to several "significant words" when understanding a sentence, we propose the Significant Semantic Strengthening (S3) strategy, which includes a novel masking and matching proxy task to promote the pre-training performance. Experiments conducted on three downstream video-text tasks and six datasets demonstrate that, we establish a new state-of-the-art in pixel-level video-text pre-training; we also achieve a satisfactory balance between the pre-training efficiency and the fine-tuning performance. The codebase are available at https://github.com/alipay/Ant-Multi-Modal-Framework/tree/main/prj/snps3_vtp. |
| title | SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks |
| topic | Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2401.17773 |