EuroBERT: Scaling Multilingual Encoders for European Languages
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916072718860288 |
|---|---|
| author | Boizard, Nicolas Gisserot-Boukhlef, Hippolyte Alves, Duarte M. Martins, André Hammal, Ayoub Corro, Caio Hudelot, Céline Malherbe, Emmanuel Malaboeuf, Etienne Jourdan, Fanny Hautreux, Gabriel Alves, João Haddad, Kevin El Faysse, Manuel Peyrard, Maxime Guerreiro, Nuno M. Fernandes, Patrick Rei, Ricardo Colombo, Pierre |
| author_facet | Boizard, Nicolas Gisserot-Boukhlef, Hippolyte Alves, Duarte M. Martins, André Hammal, Ayoub Corro, Caio Hudelot, Céline Malherbe, Emmanuel Malaboeuf, Etienne Jourdan, Fanny Hautreux, Gabriel Alves, João Haddad, Kevin El Faysse, Manuel Peyrard, Maxime Guerreiro, Nuno M. Fernandes, Patrick Rei, Ricardo Colombo, Pierre |
| contents | General-purpose multilingual vector representations, used in retrieval, regression and classification, are traditionally obtained from bidirectional encoder models. Despite their wide applicability, encoders have been recently overshadowed by advances in generative decoder-only models. However, many innovations driving this progress are not inherently tied to decoders. In this paper, we revisit the development of multilingual encoders through the lens of these advances, and introduce EuroBERT, a family of multilingual encoders covering European and widely spoken global languages. Our models outperform existing alternatives across a diverse range of tasks, spanning multilingual capabilities, mathematics, and coding, and natively supporting sequences of up to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering insights into our dataset composition and training pipeline. We publicly release the EuroBERT models, including intermediate training checkpoints, together with our training framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_05500 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EuroBERT: Scaling Multilingual Encoders for European Languages Boizard, Nicolas Gisserot-Boukhlef, Hippolyte Alves, Duarte M. Martins, André Hammal, Ayoub Corro, Caio Hudelot, Céline Malherbe, Emmanuel Malaboeuf, Etienne Jourdan, Fanny Hautreux, Gabriel Alves, João Haddad, Kevin El Faysse, Manuel Peyrard, Maxime Guerreiro, Nuno M. Fernandes, Patrick Rei, Ricardo Colombo, Pierre Computation and Language Artificial Intelligence General-purpose multilingual vector representations, used in retrieval, regression and classification, are traditionally obtained from bidirectional encoder models. Despite their wide applicability, encoders have been recently overshadowed by advances in generative decoder-only models. However, many innovations driving this progress are not inherently tied to decoders. In this paper, we revisit the development of multilingual encoders through the lens of these advances, and introduce EuroBERT, a family of multilingual encoders covering European and widely spoken global languages. Our models outperform existing alternatives across a diverse range of tasks, spanning multilingual capabilities, mathematics, and coding, and natively supporting sequences of up to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering insights into our dataset composition and training pipeline. We publicly release the EuroBERT models, including intermediate training checkpoints, together with our training framework. |
| title | EuroBERT: Scaling Multilingual Encoders for European Languages |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.05500 |