Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911779687235584 |
|---|---|
| author | Zhang, Yuanchi Wang, Yile Liu, Zijun Wang, Shuo Wang, Xiaolong Li, Peng Sun, Maosong Liu, Yang |
| author_facet | Zhang, Yuanchi Wang, Yile Liu, Zijun Wang, Shuo Wang, Xiaolong Li, Peng Sun, Maosong Liu, Yang |
| contents | While large language models (LLMs) have been pre-trained on multilingual corpora, their performance still lags behind in most languages compared to a few resource-rich languages. One common approach to mitigate this issue is to translate training data from resource-rich languages into other languages and then continue training. However, using the data obtained solely relying on translation while ignoring the original capabilities of LLMs across languages is not always effective, which we show will limit the performance of cross-lingual knowledge transfer. In this work, we propose SDRRL, a method based on Self-Distillation from Resource-Rich Languages that effectively improve multilingual performance by leveraging the internal capabilities of LLMs on resource-rich languages. We evaluate on different LLMs (LLaMA-2 and SeaLLM) and source languages across various comprehension and generation tasks, experimental results demonstrate that SDRRL can significantly enhance multilingual capabilities while minimizing the impact on original performance in resource-rich languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_12204 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages Zhang, Yuanchi Wang, Yile Liu, Zijun Wang, Shuo Wang, Xiaolong Li, Peng Sun, Maosong Liu, Yang Computation and Language While large language models (LLMs) have been pre-trained on multilingual corpora, their performance still lags behind in most languages compared to a few resource-rich languages. One common approach to mitigate this issue is to translate training data from resource-rich languages into other languages and then continue training. However, using the data obtained solely relying on translation while ignoring the original capabilities of LLMs across languages is not always effective, which we show will limit the performance of cross-lingual knowledge transfer. In this work, we propose SDRRL, a method based on Self-Distillation from Resource-Rich Languages that effectively improve multilingual performance by leveraging the internal capabilities of LLMs on resource-rich languages. We evaluate on different LLMs (LLaMA-2 and SeaLLM) and source languages across various comprehension and generation tasks, experimental results demonstrate that SDRRL can significantly enhance multilingual capabilities while minimizing the impact on original performance in resource-rich languages. |
| title | Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.12204 |