The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model design
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915133911990272 |
|---|---|
| author | Snegirev, Artem Tikhonova, Maria Maksimova, Anna Fenogenova, Alena Abramov, Alexander |
| author_facet | Snegirev, Artem Tikhonova, Maria Maksimova, Anna Fenogenova, Alena Abramov, Alexander |
| contents | Embedding models play a crucial role in Natural Language Processing (NLP) by creating text embeddings used in various tasks such as information retrieval and assessing semantic text similarity. This paper focuses on research related to embedding models in the Russian language. It introduces a new Russian-focused embedding model called ru-en-RoSBERTa and the ruMTEB benchmark, the Russian version extending the Massive Text Embedding Benchmark (MTEB). Our benchmark includes seven categories of tasks, such as semantic textual similarity, text classification, reranking, and retrieval.The research also assesses a representative set of Russian and multilingual models on the proposed benchmark. The findings indicate that the new model achieves results that are on par with state-of-the-art models in Russian. We release the model ru-en-RoSBERTa, and the ruMTEB framework comes with open-source code, integration into the original framework and a public leaderboard. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_12503 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model design Snegirev, Artem Tikhonova, Maria Maksimova, Anna Fenogenova, Alena Abramov, Alexander Computation and Language Artificial Intelligence Embedding models play a crucial role in Natural Language Processing (NLP) by creating text embeddings used in various tasks such as information retrieval and assessing semantic text similarity. This paper focuses on research related to embedding models in the Russian language. It introduces a new Russian-focused embedding model called ru-en-RoSBERTa and the ruMTEB benchmark, the Russian version extending the Massive Text Embedding Benchmark (MTEB). Our benchmark includes seven categories of tasks, such as semantic textual similarity, text classification, reranking, and retrieval.The research also assesses a representative set of Russian and multilingual models on the proposed benchmark. The findings indicate that the new model achieves results that are on par with state-of-the-art models in Russian. We release the model ru-en-RoSBERTa, and the ruMTEB framework comes with open-source code, integration into the original framework and a public leaderboard. |
| title | The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model design |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2408.12503 |