mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909471290163200 |
|---|---|
| author | Weller, Orion Chang, Benjamin Yang, Eugene Yarmohammadi, Mahsa Barham, Sam MacAvaney, Sean Cohan, Arman Soldaini, Luca Van Durme, Benjamin Lawrie, Dawn |
| author_facet | Weller, Orion Chang, Benjamin Yang, Eugene Yarmohammadi, Mahsa Barham, Sam MacAvaney, Sean Cohan, Arman Soldaini, Luca Van Durme, Benjamin Lawrie, Dawn |
| contents | Retrieval systems generally focus on web-style queries that are short and underspecified. However, advances in language models have facilitated the nascent rise of retrieval models that can understand more complex queries with diverse intents. However, these efforts have focused exclusively on English; therefore, we do not yet understand how they work across languages. We introduce mFollowIR, a multilingual benchmark for measuring instruction-following ability in retrieval models. mFollowIR builds upon the TREC NeuCLIR narratives (or instructions) that span three diverse languages (Russian, Chinese, Persian) giving both query and instruction to the retrieval models. We make small changes to the narratives and isolate how well retrieval models can follow these nuanced changes. We present results for both multilingual (XX-XX) and cross-lingual (En-XX) performance. We see strong cross-lingual performance with English-based retrievers that trained using instructions, but find a notable drop in performance in the multilingual setting, indicating that more work is needed in developing data for instruction-based multilingual retrievers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_19264 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval Weller, Orion Chang, Benjamin Yang, Eugene Yarmohammadi, Mahsa Barham, Sam MacAvaney, Sean Cohan, Arman Soldaini, Luca Van Durme, Benjamin Lawrie, Dawn Information Retrieval Computation and Language Machine Learning Retrieval systems generally focus on web-style queries that are short and underspecified. However, advances in language models have facilitated the nascent rise of retrieval models that can understand more complex queries with diverse intents. However, these efforts have focused exclusively on English; therefore, we do not yet understand how they work across languages. We introduce mFollowIR, a multilingual benchmark for measuring instruction-following ability in retrieval models. mFollowIR builds upon the TREC NeuCLIR narratives (or instructions) that span three diverse languages (Russian, Chinese, Persian) giving both query and instruction to the retrieval models. We make small changes to the narratives and isolate how well retrieval models can follow these nuanced changes. We present results for both multilingual (XX-XX) and cross-lingual (En-XX) performance. We see strong cross-lingual performance with English-based retrievers that trained using instructions, but find a notable drop in performance in the multilingual setting, indicating that more work is needed in developing data for instruction-based multilingual retrievers. |
| title | mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval |
| topic | Information Retrieval Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2501.19264 |