Cover Image

MMTEB: Massive Multilingual Text Embedding Benchmark

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Enevoldsen, Kenneth, Chung, Isaac, Kerboua, Imene, Kardos, Márton, Mathur, Ashwin, Stap, David, Gala, Jay, Siblini, Wissam, Krzemiński, Dominik, Winata, Genta Indra, Sturua, Saba, Utpala, Saiteja, Ciancone, Mathieu, Schaeffer, Marion, Sequeira, Gabriel, Misra, Diganta, Dhakal, Shreeya, Rystrøm, Jonathan, Solomatin, Roman, Çağatan, Ömer, Kundu, Akash, Bernstorff, Martin, Xiao, Shitao, Sukhlecha, Akshita, Pahwa, Bhavish, Poświata, Rafał, GV, Kranthi Kiran, Ashraf, Shawon, Auras, Daniel, Plüster, Björn, Harries, Jan Philipp, Magne, Loïc, Mohr, Isabelle, Hendriksen, Mariya, Zhu, Dawei, Gisserot-Boukhlef, Hippolyte, Aarsen, Tom, Kostkan, Jan, Wojtasik, Konrad, Lee, Taemin, Šuppa, Marek, Zhang, Crystina, Rocca, Roberta, Hamdy, Mohammed, Michail, Andrianos, Yang, John, Faysse, Manuel, Vatolin, Aleksei, Thakur, Nandan, Dey, Manan, Vasani, Dipam, Chitale, Pranjal, Tedeschi, Simone, Tai, Nguyen, Snegirev, Artem, Günther, Michael, Xia, Mengzhou, Shi, Weijia, Lù, Xing Han, Clive, Jordan, Krishnakumar, Gayatri, Maksimova, Anna, Wehrli, Silvan, Tikhonova, Maria, Panchal, Henil, Abramov, Aleksandr, Ostendorff, Malte, Liu, Zheng, Clematide, Simon, Miranda, Lester James, Fenogenova, Alena, Song, Guangyu, Safi, Ruqiya Bin, Li, Wen-Ding, Borghini, Alessia, Cassano, Federico, Su, Hongjin, Lin, Jimmy, Yen, Howard, Hansen, Lasse, Hooker, Sara, Xiao, Chenghao, Adlakha, Vaibhav, Weller, Orion, Reddy, Siva, Muennighoff, Niklas
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence Information Retrieval
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866912705738178560
author	Enevoldsen, Kenneth Chung, Isaac Kerboua, Imene Kardos, Márton Mathur, Ashwin Stap, David Gala, Jay Siblini, Wissam Krzemiński, Dominik Winata, Genta Indra Sturua, Saba Utpala, Saiteja Ciancone, Mathieu Schaeffer, Marion Sequeira, Gabriel Misra, Diganta Dhakal, Shreeya Rystrøm, Jonathan Solomatin, Roman Çağatan, Ömer Kundu, Akash Bernstorff, Martin Xiao, Shitao Sukhlecha, Akshita Pahwa, Bhavish Poświata, Rafał GV, Kranthi Kiran Ashraf, Shawon Auras, Daniel Plüster, Björn Harries, Jan Philipp Magne, Loïc Mohr, Isabelle Hendriksen, Mariya Zhu, Dawei Gisserot-Boukhlef, Hippolyte Aarsen, Tom Kostkan, Jan Wojtasik, Konrad Lee, Taemin Šuppa, Marek Zhang, Crystina Rocca, Roberta Hamdy, Mohammed Michail, Andrianos Yang, John Faysse, Manuel Vatolin, Aleksei Thakur, Nandan Dey, Manan Vasani, Dipam Chitale, Pranjal Tedeschi, Simone Tai, Nguyen Snegirev, Artem Günther, Michael Xia, Mengzhou Shi, Weijia Lù, Xing Han Clive, Jordan Krishnakumar, Gayatri Maksimova, Anna Wehrli, Silvan Tikhonova, Maria Panchal, Henil Abramov, Aleksandr Ostendorff, Malte Liu, Zheng Clematide, Simon Miranda, Lester James Fenogenova, Alena Song, Guangyu Safi, Ruqiya Bin Li, Wen-Ding Borghini, Alessia Cassano, Federico Su, Hongjin Lin, Jimmy Yen, Howard Hansen, Lasse Hooker, Sara Xiao, Chenghao Adlakha, Vaibhav Weller, Orion Reddy, Siva Muennighoff, Niklas
author_facet	Enevoldsen, Kenneth Chung, Isaac Kerboua, Imene Kardos, Márton Mathur, Ashwin Stap, David Gala, Jay Siblini, Wissam Krzemiński, Dominik Winata, Genta Indra Sturua, Saba Utpala, Saiteja Ciancone, Mathieu Schaeffer, Marion Sequeira, Gabriel Misra, Diganta Dhakal, Shreeya Rystrøm, Jonathan Solomatin, Roman Çağatan, Ömer Kundu, Akash Bernstorff, Martin Xiao, Shitao Sukhlecha, Akshita Pahwa, Bhavish Poświata, Rafał GV, Kranthi Kiran Ashraf, Shawon Auras, Daniel Plüster, Björn Harries, Jan Philipp Magne, Loïc Mohr, Isabelle Hendriksen, Mariya Zhu, Dawei Gisserot-Boukhlef, Hippolyte Aarsen, Tom Kostkan, Jan Wojtasik, Konrad Lee, Taemin Šuppa, Marek Zhang, Crystina Rocca, Roberta Hamdy, Mohammed Michail, Andrianos Yang, John Faysse, Manuel Vatolin, Aleksei Thakur, Nandan Dey, Manan Vasani, Dipam Chitale, Pranjal Tedeschi, Simone Tai, Nguyen Snegirev, Artem Günther, Michael Xia, Mengzhou Shi, Weijia Lù, Xing Han Clive, Jordan Krishnakumar, Gayatri Maksimova, Anna Wehrli, Silvan Tikhonova, Maria Panchal, Henil Abramov, Aleksandr Ostendorff, Malte Liu, Zheng Clematide, Simon Miranda, Lester James Fenogenova, Alena Song, Guangyu Safi, Ruqiya Bin Li, Wen-Ding Borghini, Alessia Cassano, Federico Su, Hongjin Lin, Jimmy Yen, Howard Hansen, Lasse Hooker, Sara Xiao, Chenghao Adlakha, Vaibhav Weller, Orion Reddy, Siva Muennighoff, Niklas
contents	Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more comprehensive evaluation, we introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) - a large-scale, community-driven expansion of MTEB, covering over 500 quality-controlled evaluation tasks across 250+ languages. MMTEB includes a diverse set of challenging, novel tasks such as instruction following, long-document retrieval, and code retrieval, representing the largest multilingual collection of evaluation tasks for embedding models to date. Using this collection, we develop several highly multilingual benchmarks, which we use to evaluate a representative set of models. We find that while large language models (LLMs) with billions of parameters can achieve state-of-the-art performance on certain language subsets and task categories, the best-performing publicly available model is multilingual-e5-large-instruct with only 560 million parameters. To facilitate accessibility and reduce computational cost, we introduce a novel downsampling method based on inter-task correlation, ensuring a diverse selection while preserving relative model rankings. Furthermore, we optimize tasks such as retrieval by sampling hard negatives, creating smaller but effective splits. These optimizations allow us to introduce benchmarks that drastically reduce computational demands. For instance, our newly introduced zero-shot English benchmark maintains a ranking order similar to the full-scale version but at a fraction of the computational cost.
format	Preprint
id	arxiv_https___arxiv_org_abs_2502_13595
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	MMTEB: Massive Multilingual Text Embedding Benchmark Enevoldsen, Kenneth Chung, Isaac Kerboua, Imene Kardos, Márton Mathur, Ashwin Stap, David Gala, Jay Siblini, Wissam Krzemiński, Dominik Winata, Genta Indra Sturua, Saba Utpala, Saiteja Ciancone, Mathieu Schaeffer, Marion Sequeira, Gabriel Misra, Diganta Dhakal, Shreeya Rystrøm, Jonathan Solomatin, Roman Çağatan, Ömer Kundu, Akash Bernstorff, Martin Xiao, Shitao Sukhlecha, Akshita Pahwa, Bhavish Poświata, Rafał GV, Kranthi Kiran Ashraf, Shawon Auras, Daniel Plüster, Björn Harries, Jan Philipp Magne, Loïc Mohr, Isabelle Hendriksen, Mariya Zhu, Dawei Gisserot-Boukhlef, Hippolyte Aarsen, Tom Kostkan, Jan Wojtasik, Konrad Lee, Taemin Šuppa, Marek Zhang, Crystina Rocca, Roberta Hamdy, Mohammed Michail, Andrianos Yang, John Faysse, Manuel Vatolin, Aleksei Thakur, Nandan Dey, Manan Vasani, Dipam Chitale, Pranjal Tedeschi, Simone Tai, Nguyen Snegirev, Artem Günther, Michael Xia, Mengzhou Shi, Weijia Lù, Xing Han Clive, Jordan Krishnakumar, Gayatri Maksimova, Anna Wehrli, Silvan Tikhonova, Maria Panchal, Henil Abramov, Aleksandr Ostendorff, Malte Liu, Zheng Clematide, Simon Miranda, Lester James Fenogenova, Alena Song, Guangyu Safi, Ruqiya Bin Li, Wen-Ding Borghini, Alessia Cassano, Federico Su, Hongjin Lin, Jimmy Yen, Howard Hansen, Lasse Hooker, Sara Xiao, Chenghao Adlakha, Vaibhav Weller, Orion Reddy, Siva Muennighoff, Niklas Computation and Language Artificial Intelligence Information Retrieval Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more comprehensive evaluation, we introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) - a large-scale, community-driven expansion of MTEB, covering over 500 quality-controlled evaluation tasks across 250+ languages. MMTEB includes a diverse set of challenging, novel tasks such as instruction following, long-document retrieval, and code retrieval, representing the largest multilingual collection of evaluation tasks for embedding models to date. Using this collection, we develop several highly multilingual benchmarks, which we use to evaluate a representative set of models. We find that while large language models (LLMs) with billions of parameters can achieve state-of-the-art performance on certain language subsets and task categories, the best-performing publicly available model is multilingual-e5-large-instruct with only 560 million parameters. To facilitate accessibility and reduce computational cost, we introduce a novel downsampling method based on inter-task correlation, ensuring a diverse selection while preserving relative model rankings. Furthermore, we optimize tasks such as retrieval by sampling hard negatives, creating smaller but effective splits. These optimizations allow us to introduce benchmarks that drastically reduce computational demands. For instance, our newly introduced zero-shot English benchmark maintains a ranking order similar to the full-scale version but at a fraction of the computational cost.
title	MMTEB: Massive Multilingual Text Embedding Benchmark
topic	Computation and Language Artificial Intelligence Information Retrieval
url	https://arxiv.org/abs/2502.13595