Composed Video Retrieval via Enriched Context and Discriminative Embeddings
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913282504261632 |
|---|---|
| author | Thawakar, Omkar Naseer, Muzammal Anwer, Rao Muhammad Khan, Salman Felsberg, Michael Shah, Mubarak Khan, Fahad Shahbaz |
| author_facet | Thawakar, Omkar Naseer, Muzammal Anwer, Rao Muhammad Khan, Salman Felsberg, Michael Shah, Mubarak Khan, Fahad Shahbaz |
| contents | Composed video retrieval (CoVR) is a challenging problem in computer vision which has recently highlighted the integration of modification text with visual queries for more sophisticated video search in large databases. Existing works predominantly rely on visual queries combined with modification text to distinguish relevant videos. However, such a strategy struggles to fully preserve the rich query-specific context in retrieved target videos and only represents the target video using visual embedding. We introduce a novel CoVR framework that leverages detailed language descriptions to explicitly encode query-specific contextual information and learns discriminative embeddings of vision only, text only and vision-text for better alignment to accurately retrieve matched target videos. Our proposed framework can be flexibly employed for both composed video (CoVR) and image (CoIR) retrieval tasks. Experiments on three datasets show that our approach obtains state-of-the-art performance for both CovR and zero-shot CoIR tasks, achieving gains as high as around 7% in terms of recall@K=1 score. Our code, models, detailed language descriptions for WebViD-CoVR dataset are available at \url{https://github.com/OmkarThawakar/composed-video-retrieval} |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_16997 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Composed Video Retrieval via Enriched Context and Discriminative Embeddings Thawakar, Omkar Naseer, Muzammal Anwer, Rao Muhammad Khan, Salman Felsberg, Michael Shah, Mubarak Khan, Fahad Shahbaz Computer Vision and Pattern Recognition Composed video retrieval (CoVR) is a challenging problem in computer vision which has recently highlighted the integration of modification text with visual queries for more sophisticated video search in large databases. Existing works predominantly rely on visual queries combined with modification text to distinguish relevant videos. However, such a strategy struggles to fully preserve the rich query-specific context in retrieved target videos and only represents the target video using visual embedding. We introduce a novel CoVR framework that leverages detailed language descriptions to explicitly encode query-specific contextual information and learns discriminative embeddings of vision only, text only and vision-text for better alignment to accurately retrieve matched target videos. Our proposed framework can be flexibly employed for both composed video (CoVR) and image (CoIR) retrieval tasks. Experiments on three datasets show that our approach obtains state-of-the-art performance for both CovR and zero-shot CoIR tasks, achieving gains as high as around 7% in terms of recall@K=1 score. Our code, models, detailed language descriptions for WebViD-CoVR dataset are available at \url{https://github.com/OmkarThawakar/composed-video-retrieval} |
| title | Composed Video Retrieval via Enriched Context and Discriminative Embeddings |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.16997 |