Multilingual Encoder Knows more than You Realize: Shared Weights Pretraining for Extremely Low-Resource Languages
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915311051079680 |
|---|---|
| author | Su, Zeli Zhang, Ziyin Xu, Guixian Liu, Jianing Han, XU Zhang, Ting Dong, Yushuang |
| author_facet | Su, Zeli Zhang, Ziyin Xu, Guixian Liu, Jianing Han, XU Zhang, Ting Dong, Yushuang |
| contents | While multilingual language models like XLM-R have advanced multilingualism in NLP, they still perform poorly in extremely low-resource languages. This situation is exacerbated by the fact that modern LLMs such as LLaMA and Qwen support far fewer languages than XLM-R, making text generation models non-existent for many languages in the world. To tackle this challenge, we propose a novel framework for adapting multilingual encoders to text generation in extremely low-resource languages. By reusing the weights between the encoder and the decoder, our framework allows the model to leverage the learned semantic space of the encoder, enabling efficient learning and effective generalization in low-resource languages. Applying this framework to four Chinese minority languages, we present XLM-SWCM, and demonstrate its superior performance on various downstream tasks even when compared with much larger models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_10852 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multilingual Encoder Knows more than You Realize: Shared Weights Pretraining for Extremely Low-Resource Languages Su, Zeli Zhang, Ziyin Xu, Guixian Liu, Jianing Han, XU Zhang, Ting Dong, Yushuang Computation and Language Artificial Intelligence While multilingual language models like XLM-R have advanced multilingualism in NLP, they still perform poorly in extremely low-resource languages. This situation is exacerbated by the fact that modern LLMs such as LLaMA and Qwen support far fewer languages than XLM-R, making text generation models non-existent for many languages in the world. To tackle this challenge, we propose a novel framework for adapting multilingual encoders to text generation in extremely low-resource languages. By reusing the weights between the encoder and the decoder, our framework allows the model to leverage the learned semantic space of the encoder, enabling efficient learning and effective generalization in low-resource languages. Applying this framework to four Chinese minority languages, we present XLM-SWCM, and demonstrate its superior performance on various downstream tasks even when compared with much larger models. |
| title | Multilingual Encoder Knows more than You Realize: Shared Weights Pretraining for Extremely Low-Resource Languages |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.10852 |