EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915652344741888 |
|---|---|
| author | Ji, Shaoxiong Li, Zihao Paavola, Jaakko Lin, Peiqin Chen, Pinzhen O'Brien, Dayyán Luo, Hengyu Schütze, Hinrich Tiedemann, Jörg Haddow, Barry |
| author_facet | Ji, Shaoxiong Li, Zihao Paavola, Jaakko Lin, Peiqin Chen, Pinzhen O'Brien, Dayyán Luo, Hengyu Schütze, Hinrich Tiedemann, Jörg Haddow, Barry |
| contents | In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, focusing on improving language coverage for low-resource languages. To facilitate continual pre-training, we compile the MaLA corpus, a comprehensive multilingual dataset enriched with curated datasets across diverse domains. Leveraging this corpus, we conduct extensive continual pre-training of the Llama 2 7B model, resulting in EMMA-500, which demonstrates robust performance across a wide collection of benchmarks, including a comprehensive set of multilingual tasks. Our results highlight the effectiveness of continual pre-training in expanding large language models' language capacity, particularly for underrepresented languages, demonstrating significant gains in cross-lingual transfer, task generalization, and language adaptability. We release the MaLA corpus, EMMA-500 model weights, scripts, and model generations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_17892 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models Ji, Shaoxiong Li, Zihao Paavola, Jaakko Lin, Peiqin Chen, Pinzhen O'Brien, Dayyán Luo, Hengyu Schütze, Hinrich Tiedemann, Jörg Haddow, Barry Computation and Language In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, focusing on improving language coverage for low-resource languages. To facilitate continual pre-training, we compile the MaLA corpus, a comprehensive multilingual dataset enriched with curated datasets across diverse domains. Leveraging this corpus, we conduct extensive continual pre-training of the Llama 2 7B model, resulting in EMMA-500, which demonstrates robust performance across a wide collection of benchmarks, including a comprehensive set of multilingual tasks. Our results highlight the effectiveness of continual pre-training in expanding large language models' language capacity, particularly for underrepresented languages, demonstrating significant gains in cross-lingual transfer, task generalization, and language adaptability. We release the MaLA corpus, EMMA-500 model weights, scripts, and model generations. |
| title | EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.17892 |