Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challenges
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910117633458176 |
|---|---|
| author | Ye, Yangfan Feng, Xiaocheng Feng, Xiachong Qin, Libo Huang, Yichong Huang, Lei Ma, Weitao Hong, Qichen Zhang, Zhirui Lu, Yunfei Yan, Xiaohui Tang, Duyu Tu, Dandan Qin, Bing |
| author_facet | Ye, Yangfan Feng, Xiaocheng Feng, Xiachong Qin, Libo Huang, Yichong Huang, Lei Ma, Weitao Hong, Qichen Zhang, Zhirui Lu, Yunfei Yan, Xiaohui Tang, Duyu Tu, Dandan Qin, Bing |
| contents | Current large language models (LLMs) often exhibit imbalances in multilingual capabilities and cultural adaptability, largely attributed to their English-centric pre-training data. In this paper, we introduce and investigate cross-lingual latent transplantation (XTransplant), a probing framework which aims to further exploit the model's internalized multilingual knowledge during inference and examine its effects on the multilingual capability and cultural adaptability of LLMs. XTransplant framework enables models to harness the complementary strengths of both English and non-English resources by transplanting latent activations across languages. Through extensive analysis, we empirically demonstrate that XTransplant, a form of cross-lingual interaction, has mutually beneficial effects on the multilingual capability and cultural adaptability of LLMs, particularly for low-resource languages and cultures. We further reveal that attention modules play a pivotal role in supporting multilingual understanding, while feed-forward modules are more adept at capturing culture-specific knowledge. In addition, we conduct in-depth analysis of XTransplant's stability, effectiveness, and generalizability. By probing the upper bound performance of XTransplant, we expose the considerable underutilization of current LLMs' multilingual potential-a challenge that remains open. We hope our analysis offers a new lens for advancing cross-lingual interactions and better leveraging models' internalized multilingual knowledge. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_12686 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challenges Ye, Yangfan Feng, Xiaocheng Feng, Xiachong Qin, Libo Huang, Yichong Huang, Lei Ma, Weitao Hong, Qichen Zhang, Zhirui Lu, Yunfei Yan, Xiaohui Tang, Duyu Tu, Dandan Qin, Bing Computation and Language Current large language models (LLMs) often exhibit imbalances in multilingual capabilities and cultural adaptability, largely attributed to their English-centric pre-training data. In this paper, we introduce and investigate cross-lingual latent transplantation (XTransplant), a probing framework which aims to further exploit the model's internalized multilingual knowledge during inference and examine its effects on the multilingual capability and cultural adaptability of LLMs. XTransplant framework enables models to harness the complementary strengths of both English and non-English resources by transplanting latent activations across languages. Through extensive analysis, we empirically demonstrate that XTransplant, a form of cross-lingual interaction, has mutually beneficial effects on the multilingual capability and cultural adaptability of LLMs, particularly for low-resource languages and cultures. We further reveal that attention modules play a pivotal role in supporting multilingual understanding, while feed-forward modules are more adept at capturing culture-specific knowledge. In addition, we conduct in-depth analysis of XTransplant's stability, effectiveness, and generalizability. By probing the upper bound performance of XTransplant, we expose the considerable underutilization of current LLMs' multilingual potential-a challenge that remains open. We hope our analysis offers a new lens for advancing cross-lingual interactions and better leveraging models' internalized multilingual knowledge. |
| title | Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challenges |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.12686 |