Multi-modal Fusion and Query Refinement Network for Video Moment Retrieval and Highlight Detection
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916571743518720 |
|---|---|
| author | Xu, Yifang Sun, Yunzhuo Zhai, Benxiang Xie, Zien Jia, Youyao Du, Sidan |
| author_facet | Xu, Yifang Sun, Yunzhuo Zhai, Benxiang Xie, Zien Jia, Youyao Du, Sidan |
| contents | Given a video and a linguistic query, video moment retrieval and highlight detection (MR&HD) aim to locate all the relevant spans while simultaneously predicting saliency scores. Most existing methods utilize RGB images as input, overlooking the inherent multi-modal visual signals like optical flow and depth. In this paper, we propose a Multi-modal Fusion and Query Refinement Network (MRNet) to learn complementary information from multi-modal cues. Specifically, we design a multi-modal fusion module to dynamically combine RGB, optical flow, and depth map. Furthermore, to simulate human understanding of sentences, we introduce a query refinement module that merges text at different granularities, containing word-, phrase-, and sentence-wise levels. Comprehensive experiments on QVHighlights and Charades datasets indicate that MRNet outperforms current state-of-the-art methods, achieving notable improvements in MR-mAP@Avg (+3.41) and HD-HIT@1 (+3.46) on QVHighlights. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_10692 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multi-modal Fusion and Query Refinement Network for Video Moment Retrieval and Highlight Detection Xu, Yifang Sun, Yunzhuo Zhai, Benxiang Xie, Zien Jia, Youyao Du, Sidan Computer Vision and Pattern Recognition Given a video and a linguistic query, video moment retrieval and highlight detection (MR&HD) aim to locate all the relevant spans while simultaneously predicting saliency scores. Most existing methods utilize RGB images as input, overlooking the inherent multi-modal visual signals like optical flow and depth. In this paper, we propose a Multi-modal Fusion and Query Refinement Network (MRNet) to learn complementary information from multi-modal cues. Specifically, we design a multi-modal fusion module to dynamically combine RGB, optical flow, and depth map. Furthermore, to simulate human understanding of sentences, we introduce a query refinement module that merges text at different granularities, containing word-, phrase-, and sentence-wise levels. Comprehensive experiments on QVHighlights and Charades datasets indicate that MRNet outperforms current state-of-the-art methods, achieving notable improvements in MR-mAP@Avg (+3.41) and HD-HIT@1 (+3.46) on QVHighlights. |
| title | Multi-modal Fusion and Query Refinement Network for Video Moment Retrieval and Highlight Detection |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.10692 |