Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909553457627136 |
|---|---|
| author | Hur, Chan Hong, Jeong-hun Lee, Dong-hun Kang, Dabin Myeong, Semin Park, Sang-hyo Park, Hyeyoung |
| author_facet | Hur, Chan Hong, Jeong-hun Lee, Dong-hun Kang, Dabin Myeong, Semin Park, Sang-hyo Park, Hyeyoung |
| contents | In recent text-video retrieval, the use of additional captions from vision-language models has shown promising effects on the performance. However, existing models using additional captions often have struggled to capture the rich semantics, including temporal changes, inherent in the video. In addition, incorrect information caused by generative models can lead to inaccurate retrieval. To address these issues, we propose a new framework, Narrating the Video (NarVid), which strategically leverages the comprehensive information available from frame-level captions, the narration. The proposed NarVid exploits narration in multiple ways: 1) feature enhancement through cross-modal interactions between narration and video, 2) query-aware adaptive filtering to suppress irrelevant or incorrect information, 3) dual-modal matching score by adding query-video similarity and query-narration similarity, and 4) hard-negative loss to learn discriminative features from multiple perspectives using the two similarities from different views. Experimental results demonstrate that NarVid achieves state-of-the-art performance on various benchmark datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_05186 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions Hur, Chan Hong, Jeong-hun Lee, Dong-hun Kang, Dabin Myeong, Semin Park, Sang-hyo Park, Hyeyoung Computer Vision and Pattern Recognition In recent text-video retrieval, the use of additional captions from vision-language models has shown promising effects on the performance. However, existing models using additional captions often have struggled to capture the rich semantics, including temporal changes, inherent in the video. In addition, incorrect information caused by generative models can lead to inaccurate retrieval. To address these issues, we propose a new framework, Narrating the Video (NarVid), which strategically leverages the comprehensive information available from frame-level captions, the narration. The proposed NarVid exploits narration in multiple ways: 1) feature enhancement through cross-modal interactions between narration and video, 2) query-aware adaptive filtering to suppress irrelevant or incorrect information, 3) dual-modal matching score by adding query-video similarity and query-narration similarity, and 4) hard-negative loss to learn discriminative features from multiple perspectives using the two similarities from different views. Experimental results demonstrate that NarVid achieves state-of-the-art performance on various benchmark datasets. |
| title | Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.05186 |