Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914940765339648 |
|---|---|
| author | Xiong, Yuanhao Zhao, Long Gong, Boqing Yang, Ming-Hsuan Schroff, Florian Liu, Ting Hsieh, Cho-Jui Yuan, Liangzhe |
| author_facet | Xiong, Yuanhao Zhao, Long Gong, Boqing Yang, Ming-Hsuan Schroff, Florian Liu, Ting Hsieh, Cho-Jui Yuan, Liangzhe |
| contents | Existing video-language pre-training methods primarily focus on instance-level alignment between video clips and captions via global contrastive learning but neglect rich fine-grained local information in both videos and text, which is of importance to downstream tasks requiring temporal localization and semantic reasoning. A powerful model is expected to be capable of capturing region-object correspondences and recognizing scene changes in a video clip, reflecting spatial and temporal granularity, respectively. To strengthen model's understanding into such fine-grained details, we propose a simple yet effective video-language modeling framework, S-ViLM, by exploiting the intrinsic structures of these two modalities. It includes two novel designs, inter-clip spatial grounding and intra-clip temporal grouping, to promote learning region-object alignment and temporal-aware features, simultaneously. Comprehensive evaluations demonstrate that S-ViLM performs favorably against existing approaches in learning more expressive representations. Specifically, S-ViLM surpasses the state-of-the-art methods substantially on four representative downstream tasks, covering text-video retrieval, video question answering, video action recognition, and temporal action localization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2303_16341 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding Xiong, Yuanhao Zhao, Long Gong, Boqing Yang, Ming-Hsuan Schroff, Florian Liu, Ting Hsieh, Cho-Jui Yuan, Liangzhe Computer Vision and Pattern Recognition Existing video-language pre-training methods primarily focus on instance-level alignment between video clips and captions via global contrastive learning but neglect rich fine-grained local information in both videos and text, which is of importance to downstream tasks requiring temporal localization and semantic reasoning. A powerful model is expected to be capable of capturing region-object correspondences and recognizing scene changes in a video clip, reflecting spatial and temporal granularity, respectively. To strengthen model's understanding into such fine-grained details, we propose a simple yet effective video-language modeling framework, S-ViLM, by exploiting the intrinsic structures of these two modalities. It includes two novel designs, inter-clip spatial grounding and intra-clip temporal grouping, to promote learning region-object alignment and temporal-aware features, simultaneously. Comprehensive evaluations demonstrate that S-ViLM performs favorably against existing approaches in learning more expressive representations. Specifically, S-ViLM surpasses the state-of-the-art methods substantially on four representative downstream tasks, covering text-video retrieval, video question answering, video action recognition, and temporal action localization. |
| title | Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2303.16341 |