Lens: Rethinking Multilingual Enhancement for Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916758150971392 |
|---|---|
| author | Zhao, Weixiang Hu, Yulin Guo, Jiahe Sui, Xingyu Wu, Tongtong Deng, Yang Zhao, Yanyan Qin, Bing Che, Wanxiang Liu, Ting |
| author_facet | Zhao, Weixiang Hu, Yulin Guo, Jiahe Sui, Xingyu Wu, Tongtong Deng, Yang Zhao, Yanyan Qin, Bing Che, Wanxiang Liu, Ting |
| contents | As global demand for multilingual large language models (LLMs) grows, most LLMs still remain overly focused on English, leading to the limited access to advanced AI for non-English speakers. Current methods to enhance multilingual capabilities largely rely on data-driven post-training techniques, such as multilingual instruction tuning or continual pre-training. However, these approaches exhibit significant limitations, including high resource cost, exacerbation of off-target issue and catastrophic forgetting of central language abilities. To this end, we propose Lens, a novel approach that enhances multilingual capabilities by leveraging LLMs' internal language representation spaces. Lens operates on two subspaces: the language-agnostic subspace, where it aligns target languages with the central language to inherit strong semantic representations, and the language-specific subspace, where it separates target and central languages to preserve linguistic specificity. Experiments on three English-centric LLMs show that Lens significantly improves multilingual performance while maintaining the model's English proficiency, achieving better results with less computational cost compared to existing post-training approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_04407 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Lens: Rethinking Multilingual Enhancement for Large Language Models Zhao, Weixiang Hu, Yulin Guo, Jiahe Sui, Xingyu Wu, Tongtong Deng, Yang Zhao, Yanyan Qin, Bing Che, Wanxiang Liu, Ting Computation and Language As global demand for multilingual large language models (LLMs) grows, most LLMs still remain overly focused on English, leading to the limited access to advanced AI for non-English speakers. Current methods to enhance multilingual capabilities largely rely on data-driven post-training techniques, such as multilingual instruction tuning or continual pre-training. However, these approaches exhibit significant limitations, including high resource cost, exacerbation of off-target issue and catastrophic forgetting of central language abilities. To this end, we propose Lens, a novel approach that enhances multilingual capabilities by leveraging LLMs' internal language representation spaces. Lens operates on two subspaces: the language-agnostic subspace, where it aligns target languages with the central language to inherit strong semantic representations, and the language-specific subspace, where it separates target and central languages to preserve linguistic specificity. Experiments on three English-centric LLMs show that Lens significantly improves multilingual performance while maintaining the model's English proficiency, achieving better results with less computational cost compared to existing post-training approaches. |
| title | Lens: Rethinking Multilingual Enhancement for Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.04407 |