Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.20698 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908355384049664 |
|---|---|
| author | Samuel, Saron DeGenaro, Dan Guallar-Blasco, Jimena Sanders, Kate Eisape, Oluwaseun Spendlove, Tanner Reddy, Arun Martin, Alexander Yates, Andrew Yang, Eugene Carpenter, Cameron Etter, David Kayi, Efsun Wiesner, Matthew Murray, Kenton Kriz, Reno |
| author_facet | Samuel, Saron DeGenaro, Dan Guallar-Blasco, Jimena Sanders, Kate Eisape, Oluwaseun Spendlove, Tanner Reddy, Arun Martin, Alexander Yates, Andrew Yang, Eugene Carpenter, Cameron Etter, David Kayi, Efsun Wiesner, Matthew Murray, Kenton Kriz, Reno |
| contents | Videos inherently contain multiple modalities, including visual events, text overlays, sounds, and speech, all of which are important for retrieval. However, state-of-the-art multimodal language models like VAST and LanguageBind are built on vision-language models (VLMs), and thus overly prioritize visual signals. Retrieval benchmarks further reinforce this bias by focusing on visual queries and neglecting other modalities. We create a search system MMMORRF that extracts text and features from both visual and audio modalities and integrates them with a novel modality-aware weighted reciprocal rank fusion. MMMORRF is both effective and efficient, demonstrating practicality in searching videos based on users' information needs instead of visual descriptive queries. We evaluate MMMORRF on MultiVENT 2.0 and TVR, two multimodal benchmarks designed for more targeted information needs, and find that it improves nDCG@20 by 81% over leading multimodal encoders and 37% over single-modality retrieval, demonstrating the value of integrating diverse modalities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_20698 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusion Samuel, Saron DeGenaro, Dan Guallar-Blasco, Jimena Sanders, Kate Eisape, Oluwaseun Spendlove, Tanner Reddy, Arun Martin, Alexander Yates, Andrew Yang, Eugene Carpenter, Cameron Etter, David Kayi, Efsun Wiesner, Matthew Murray, Kenton Kriz, Reno Computer Vision and Pattern Recognition Information Retrieval Videos inherently contain multiple modalities, including visual events, text overlays, sounds, and speech, all of which are important for retrieval. However, state-of-the-art multimodal language models like VAST and LanguageBind are built on vision-language models (VLMs), and thus overly prioritize visual signals. Retrieval benchmarks further reinforce this bias by focusing on visual queries and neglecting other modalities. We create a search system MMMORRF that extracts text and features from both visual and audio modalities and integrates them with a novel modality-aware weighted reciprocal rank fusion. MMMORRF is both effective and efficient, demonstrating practicality in searching videos based on users' information needs instead of visual descriptive queries. We evaluate MMMORRF on MultiVENT 2.0 and TVR, two multimodal benchmarks designed for more targeted information needs, and find that it improves nDCG@20 by 81% over leading multimodal encoders and 37% over single-modality retrieval, demonstrating the value of integrating diverse modalities. |
| title | MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusion |
| topic | Computer Vision and Pattern Recognition Information Retrieval |
| url | https://arxiv.org/abs/2503.20698 |