Multi-scale 2D Temporal Map Diffusion Models for Natural Language Video Localization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910298692124672 |
|---|---|
| author | Zhang, Chongzhi Zhang, Mingyuan Teng, Zhiyang Li, Jiayi Zhu, Xizhou Lu, Lewei Liu, Ziwei Sun, Aixin |
| author_facet | Zhang, Chongzhi Zhang, Mingyuan Teng, Zhiyang Li, Jiayi Zhu, Xizhou Lu, Lewei Liu, Ziwei Sun, Aixin |
| contents | Natural Language Video Localization (NLVL), grounding phrases from natural language descriptions to corresponding video segments, is a complex yet critical task in video understanding. Despite ongoing advancements, many existing solutions lack the capability to globally capture temporal dynamics of the video data. In this study, we present a novel approach to NLVL that aims to address this issue. Our method involves the direct generation of a global 2D temporal map via a conditional denoising diffusion process, based on the input video and language query. The main challenges are the inherent sparsity and discontinuity of a 2D temporal map in devising the diffusion decoder. To address these challenges, we introduce a multi-scale technique and develop an innovative diffusion decoder. Our approach effectively encapsulates the interaction between the query and video data across various time scales. Experiments on the Charades and DiDeMo datasets underscore the potency of our design. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_08232 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multi-scale 2D Temporal Map Diffusion Models for Natural Language Video Localization Zhang, Chongzhi Zhang, Mingyuan Teng, Zhiyang Li, Jiayi Zhu, Xizhou Lu, Lewei Liu, Ziwei Sun, Aixin Computer Vision and Pattern Recognition Natural Language Video Localization (NLVL), grounding phrases from natural language descriptions to corresponding video segments, is a complex yet critical task in video understanding. Despite ongoing advancements, many existing solutions lack the capability to globally capture temporal dynamics of the video data. In this study, we present a novel approach to NLVL that aims to address this issue. Our method involves the direct generation of a global 2D temporal map via a conditional denoising diffusion process, based on the input video and language query. The main challenges are the inherent sparsity and discontinuity of a 2D temporal map in devising the diffusion decoder. To address these challenges, we introduce a multi-scale technique and develop an innovative diffusion decoder. Our approach effectively encapsulates the interaction between the query and video data across various time scales. Experiments on the Charades and DiDeMo datasets underscore the potency of our design. |
| title | Multi-scale 2D Temporal Map Diffusion Models for Natural Language Video Localization |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.08232 |