Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915873507246080 |
|---|---|
| author | Omnilingual SONAR Team Janeiro, João Maria Cabot, Pere-Lluís Huguet Tsiamas, Ioannis Meng, Yen Iyer, Vivek Ramírez, Guillem Barrault, Loic Alastruey, Belen Chung, Yu-An Costa-Jussa, Marta R. Dale, David Heffernan, Kevin Jo, Jaehyeong Kozhevnikov, Artyom Mourachko, Alexandre Ropers, Christophe Schwenk, Holger Duquenne, Paul-Ambroise |
| author_facet | Omnilingual SONAR Team Janeiro, João Maria Cabot, Pere-Lluís Huguet Tsiamas, Ioannis Meng, Yen Iyer, Vivek Ramírez, Guillem Barrault, Loic Alastruey, Belen Chung, Yu-An Costa-Jussa, Marta R. Dale, David Heffernan, Kevin Jo, Jaehyeong Kozhevnikov, Artyom Mourachko, Alexandre Ropers, Christophe Schwenk, Holger Duquenne, Paul-Ambroise |
| contents | Cross-lingual sentence encoders typically cover only a few hundred languages and often trade downstream quality for stronger alignment, limiting their adoption. We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-modal sentence embedding models that natively embed text, speech, code, and mathematical expressions in a single semantic space, while delivering state-of-the-art downstream performance at the scale of thousands of languages, from high-resource to extremely low-resource varieties. To reach this scale without representation collapse, we use progressive training. We first learn a strong foundational space for 200 languages with an LLM-initialized encoder-decoder, combining token-level decoding with a novel split-softmax contrastive loss and synthetic hard negatives. Building on this foundation, we expand to several thousands language varieties via a two-stage teacher-student encoder distillation framework. Finally, we demonstrate the cross-modal extensibility of this space by seamlessly mapping 177 spoken languages into it. OmniSONAR halves cross-lingual similarity search error on the 200-language FLORES dataset and reduces error by a factor of 15 on the 1,560-language BIBLE benchmark. It also enables strong translation, outperforming NLLB-3B on multilingual benchmarks and exceeding prior models (including much larger LLMs) by 15 chrF++ points on 1,560 languages into English BIBLE translation. OmniSONAR also performs strongly on MTEB and XLCoST. For speech, OmniSONAR achieves a 43% lower similarity-search error and reaches 97% of SeamlessM4T speech-to-text quality, despite being zero-shot for translation (trained only on ASR data). Finally, by training an encoder-decoder LM, Spectrum, exclusively on English text processing OmniSONAR embedding sequences, we unlock high-performance transfer to thousands of languages and speech for complex downstream tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_16606 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech Omnilingual SONAR Team Janeiro, João Maria Cabot, Pere-Lluís Huguet Tsiamas, Ioannis Meng, Yen Iyer, Vivek Ramírez, Guillem Barrault, Loic Alastruey, Belen Chung, Yu-An Costa-Jussa, Marta R. Dale, David Heffernan, Kevin Jo, Jaehyeong Kozhevnikov, Artyom Mourachko, Alexandre Ropers, Christophe Schwenk, Holger Duquenne, Paul-Ambroise Computation and Language Cross-lingual sentence encoders typically cover only a few hundred languages and often trade downstream quality for stronger alignment, limiting their adoption. We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-modal sentence embedding models that natively embed text, speech, code, and mathematical expressions in a single semantic space, while delivering state-of-the-art downstream performance at the scale of thousands of languages, from high-resource to extremely low-resource varieties. To reach this scale without representation collapse, we use progressive training. We first learn a strong foundational space for 200 languages with an LLM-initialized encoder-decoder, combining token-level decoding with a novel split-softmax contrastive loss and synthetic hard negatives. Building on this foundation, we expand to several thousands language varieties via a two-stage teacher-student encoder distillation framework. Finally, we demonstrate the cross-modal extensibility of this space by seamlessly mapping 177 spoken languages into it. OmniSONAR halves cross-lingual similarity search error on the 200-language FLORES dataset and reduces error by a factor of 15 on the 1,560-language BIBLE benchmark. It also enables strong translation, outperforming NLLB-3B on multilingual benchmarks and exceeding prior models (including much larger LLMs) by 15 chrF++ points on 1,560 languages into English BIBLE translation. OmniSONAR also performs strongly on MTEB and XLCoST. For speech, OmniSONAR achieves a 43% lower similarity-search error and reaches 97% of SeamlessM4T speech-to-text quality, despite being zero-shot for translation (trained only on ASR data). Finally, by training an encoder-decoder LM, Spectrum, exclusively on English text processing OmniSONAR embedding sequences, we unlock high-performance transfer to thousands of languages and speech for complex downstream tasks. |
| title | Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.16606 |