BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915213471645696 |
|---|---|
| author | Su, Hongjin Yen, Howard Xia, Mengzhou Shi, Weijia Muennighoff, Niklas Wang, Han-yu Liu, Haisu Shi, Quan Siegel, Zachary S. Tang, Michael Sun, Ruoxi Yoon, Jinsung Arik, Sercan O. Chen, Danqi Yu, Tao |
| author_facet | Su, Hongjin Yen, Howard Xia, Mengzhou Shi, Weijia Muennighoff, Niklas Wang, Han-yu Liu, Haisu Shi, Quan Siegel, Zachary S. Tang, Michael Sun, Ruoxi Yoon, Jinsung Arik, Sercan O. Chen, Danqi Yu, Tao |
| contents | Existing retrieval benchmarks primarily consist of information-seeking queries (e.g., aggregated questions from search engines) where keyword or semantic-based retrieval is usually sufficient. However, many complex real-world queries require in-depth reasoning to identify relevant documents that go beyond surface form matching. For example, finding documentation for a coding question requires understanding the logic and syntax of the functions involved. To better benchmark retrieval on such challenging queries, we introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding. These queries are drawn from naturally occurring and carefully curated human data. Extensive evaluation reveals that even state-of-the-art retrieval models perform poorly on BRIGHT. The leading model on the MTEB leaderboard (Muennighoff et al., 2023) SFR-Embedding-Mistral (Meng et al., 2024), which achieves a score of 59.0 nDCG@10,1 produces a score of nDCG@10 of 18.3 on BRIGHT. We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points. Moreover, incorporating retrieved documents from the top-performing retriever boosts question-answering performance. We believe that BRIGHT paves the way for future research on retrieval systems in more realistic and challenging settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_12883 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval Su, Hongjin Yen, Howard Xia, Mengzhou Shi, Weijia Muennighoff, Niklas Wang, Han-yu Liu, Haisu Shi, Quan Siegel, Zachary S. Tang, Michael Sun, Ruoxi Yoon, Jinsung Arik, Sercan O. Chen, Danqi Yu, Tao Computation and Language Artificial Intelligence Information Retrieval Existing retrieval benchmarks primarily consist of information-seeking queries (e.g., aggregated questions from search engines) where keyword or semantic-based retrieval is usually sufficient. However, many complex real-world queries require in-depth reasoning to identify relevant documents that go beyond surface form matching. For example, finding documentation for a coding question requires understanding the logic and syntax of the functions involved. To better benchmark retrieval on such challenging queries, we introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding. These queries are drawn from naturally occurring and carefully curated human data. Extensive evaluation reveals that even state-of-the-art retrieval models perform poorly on BRIGHT. The leading model on the MTEB leaderboard (Muennighoff et al., 2023) SFR-Embedding-Mistral (Meng et al., 2024), which achieves a score of 59.0 nDCG@10,1 produces a score of nDCG@10 of 18.3 on BRIGHT. We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points. Moreover, incorporating retrieved documents from the top-performing retriever boosts question-answering performance. We believe that BRIGHT paves the way for future research on retrieval systems in more realistic and challenging settings. |
| title | BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval |
| topic | Computation and Language Artificial Intelligence Information Retrieval |
| url | https://arxiv.org/abs/2407.12883 |