Unlocking the Potential of Model Merging for Low-Resource Languages
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910817968979968 |
|---|---|
| author | Tao, Mingxu Zhang, Chen Huang, Quzhe Ma, Tianyao Huang, Songfang Zhao, Dongyan Feng, Yansong |
| author_facet | Tao, Mingxu Zhang, Chen Huang, Quzhe Ma, Tianyao Huang, Songfang Zhao, Dongyan Feng, Yansong |
| contents | Adapting large language models (LLMs) to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT). However, this CT-then-SFT approach struggles with limited data in the context of low-resource languages, failing to balance language modeling and task-solving capabilities. We thus propose model merging as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. We use model merging to develop task-solving LLMs for low-resource languages without SFT data in the target languages. Our experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data. Observing performance saturation in model merging with more training tokens, we further analyze the merging process and introduce a slack variable to the model merging algorithm to mitigate the loss of important parameters, thereby enhancing performance. We hope that model merging can benefit more human languages suffering from data scarcity with its higher data efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_03994 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Unlocking the Potential of Model Merging for Low-Resource Languages Tao, Mingxu Zhang, Chen Huang, Quzhe Ma, Tianyao Huang, Songfang Zhao, Dongyan Feng, Yansong Computation and Language Artificial Intelligence Adapting large language models (LLMs) to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT). However, this CT-then-SFT approach struggles with limited data in the context of low-resource languages, failing to balance language modeling and task-solving capabilities. We thus propose model merging as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. We use model merging to develop task-solving LLMs for low-resource languages without SFT data in the target languages. Our experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data. Observing performance saturation in model merging with more training tokens, we further analyze the merging process and introduce a slack variable to the model merging algorithm to mitigate the loss of important parameters, thereby enhancing performance. We hope that model merging can benefit more human languages suffering from data scarcity with its higher data efficiency. |
| title | Unlocking the Potential of Model Merging for Low-Resource Languages |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.03994 |