AutoTVG: A New Vision-language Pre-training Paradigm for Temporal Video Grounding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916283009728512 |
|---|---|
| author | Zhang, Xing Gu, Jiaxi Zhao, Haoyu Wang, Shicong Xu, Hang Pei, Renjing Xu, Songcen Wu, Zuxuan Jiang, Yu-Gang |
| author_facet | Zhang, Xing Gu, Jiaxi Zhao, Haoyu Wang, Shicong Xu, Hang Pei, Renjing Xu, Songcen Wu, Zuxuan Jiang, Yu-Gang |
| contents | Temporal Video Grounding (TVG) aims to localize a moment from an untrimmed video given the language description. Since the annotation of TVG is labor-intensive, TVG under limited supervision has accepted attention in recent years. The great success of vision-language pre-training guides TVG to follow the traditional "pre-training + fine-tuning" paradigm, however, the pre-training process would suffer from a lack of temporal modeling and fine-grained alignment due to the difference of data nature between pre-train and test. Besides, the large gap between pretext and downstream tasks makes zero-shot testing impossible for the pre-trained model. To avoid the drawbacks of the traditional paradigm, we propose AutoTVG, a new vision-language pre-training paradigm for TVG that enables the model to learn semantic alignment and boundary regression from automatically annotated untrimmed videos. To be specific, AutoTVG consists of a novel Captioned Moment Generation (CMG) module to generate captioned moments from untrimmed videos, and TVGNet with a regression head to predict localization results. Experimental results on Charades-STA and ActivityNet Captions show that, regarding zero-shot temporal video grounding, AutoTVG achieves highly competitive performance with in-distribution methods under out-of-distribution testing, and is superior to existing pre-training frameworks with much less training data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07091 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AutoTVG: A New Vision-language Pre-training Paradigm for Temporal Video Grounding Zhang, Xing Gu, Jiaxi Zhao, Haoyu Wang, Shicong Xu, Hang Pei, Renjing Xu, Songcen Wu, Zuxuan Jiang, Yu-Gang Computer Vision and Pattern Recognition Temporal Video Grounding (TVG) aims to localize a moment from an untrimmed video given the language description. Since the annotation of TVG is labor-intensive, TVG under limited supervision has accepted attention in recent years. The great success of vision-language pre-training guides TVG to follow the traditional "pre-training + fine-tuning" paradigm, however, the pre-training process would suffer from a lack of temporal modeling and fine-grained alignment due to the difference of data nature between pre-train and test. Besides, the large gap between pretext and downstream tasks makes zero-shot testing impossible for the pre-trained model. To avoid the drawbacks of the traditional paradigm, we propose AutoTVG, a new vision-language pre-training paradigm for TVG that enables the model to learn semantic alignment and boundary regression from automatically annotated untrimmed videos. To be specific, AutoTVG consists of a novel Captioned Moment Generation (CMG) module to generate captioned moments from untrimmed videos, and TVGNet with a regression head to predict localization results. Experimental results on Charades-STA and ActivityNet Captions show that, regarding zero-shot temporal video grounding, AutoTVG achieves highly competitive performance with in-distribution methods under out-of-distribution testing, and is superior to existing pre-training frameworks with much less training data. |
| title | AutoTVG: A New Vision-language Pre-training Paradigm for Temporal Video Grounding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.07091 |