SambaLingo: Teaching Large Language Models New Languages
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911958963322880 |
|---|---|
| author | Csaki, Zoltan Li, Bo Li, Jonathan Xu, Qiantong Pawakapan, Pian Zhang, Leon Du, Yun Zhao, Hengyu Hu, Changran Thakker, Urmish |
| author_facet | Csaki, Zoltan Li, Bo Li, Jonathan Xu, Qiantong Pawakapan, Pian Zhang, Leon Du, Yun Zhao, Hengyu Hu, Changran Thakker, Urmish |
| contents | Despite the widespread availability of LLMs, there remains a substantial gap in their capabilities and availability across diverse languages. One approach to address these issues has been to take an existing pre-trained LLM and continue to train it on new languages. While prior works have experimented with language adaptation, many questions around best practices and methodology have not been covered. In this paper, we present a comprehensive investigation into the adaptation of LLMs to new languages. Our study covers the key components in this process, including vocabulary extension, direct preference optimization and the data scarcity problem for human alignment in low-resource languages. We scale these experiments across 9 languages and 2 parameter scales (7B and 70B). We compare our models against Llama 2, Aya-101, XGLM, BLOOM and existing language experts, outperforming all prior published baselines. Additionally, all evaluation code and checkpoints are made public to facilitate future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_05829 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SambaLingo: Teaching Large Language Models New Languages Csaki, Zoltan Li, Bo Li, Jonathan Xu, Qiantong Pawakapan, Pian Zhang, Leon Du, Yun Zhao, Hengyu Hu, Changran Thakker, Urmish Computation and Language Artificial Intelligence Machine Learning Despite the widespread availability of LLMs, there remains a substantial gap in their capabilities and availability across diverse languages. One approach to address these issues has been to take an existing pre-trained LLM and continue to train it on new languages. While prior works have experimented with language adaptation, many questions around best practices and methodology have not been covered. In this paper, we present a comprehensive investigation into the adaptation of LLMs to new languages. Our study covers the key components in this process, including vocabulary extension, direct preference optimization and the data scarcity problem for human alignment in low-resource languages. We scale these experiments across 9 languages and 2 parameter scales (7B and 70B). We compare our models against Llama 2, Aya-101, XGLM, BLOOM and existing language experts, outperforming all prior published baselines. Additionally, all evaluation code and checkpoints are made public to facilitate future research. |
| title | SambaLingo: Teaching Large Language Models New Languages |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2404.05829 |