Building Russian Benchmark for Evaluation of Information Retrieval Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916694479339520 |
|---|---|
| author | Kovalev, Grigory Tikhomirov, Mikhail Kozhevnikov, Evgeny Kornilov, Max Loukachevitch, Natalia |
| author_facet | Kovalev, Grigory Tikhomirov, Mikhail Kozhevnikov, Evgeny Kornilov, Max Loukachevitch, Natalia |
| contents | We introduce RusBEIR, a comprehensive benchmark designed for zero-shot evaluation of information retrieval (IR) models in the Russian language. Comprising 17 datasets from various domains, it integrates adapted, translated, and newly created datasets, enabling systematic comparison of lexical and neural models. Our study highlights the importance of preprocessing for lexical models in morphologically rich languages and confirms BM25 as a strong baseline for full-document retrieval. Neural models, such as mE5-large and BGE-M3, demonstrate superior performance on most datasets, but face challenges with long-document retrieval due to input size constraints. RusBEIR offers a unified, open-source framework that promotes research in Russian-language information retrieval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_12879 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Building Russian Benchmark for Evaluation of Information Retrieval Models Kovalev, Grigory Tikhomirov, Mikhail Kozhevnikov, Evgeny Kornilov, Max Loukachevitch, Natalia Information Retrieval Computation and Language We introduce RusBEIR, a comprehensive benchmark designed for zero-shot evaluation of information retrieval (IR) models in the Russian language. Comprising 17 datasets from various domains, it integrates adapted, translated, and newly created datasets, enabling systematic comparison of lexical and neural models. Our study highlights the importance of preprocessing for lexical models in morphologically rich languages and confirms BM25 as a strong baseline for full-document retrieval. Neural models, such as mE5-large and BGE-M3, demonstrate superior performance on most datasets, but face challenges with long-document retrieval due to input size constraints. RusBEIR offers a unified, open-source framework that promotes research in Russian-language information retrieval. |
| title | Building Russian Benchmark for Evaluation of Information Retrieval Models |
| topic | Information Retrieval Computation and Language |
| url | https://arxiv.org/abs/2504.12879 |