Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917685878587392 |
|---|---|
| author | Tang, Tianyi Luo, Wenyang Huang, Haoyang Zhang, Dongdong Wang, Xiaolei Zhao, Xin Wei, Furu Wen, Ji-Rong |
| author_facet | Tang, Tianyi Luo, Wenyang Huang, Haoyang Zhang, Dongdong Wang, Xiaolei Zhao, Xin Wei, Furu Wen, Ji-Rong |
| contents | Large language models (LLMs) demonstrate remarkable multilingual capabilities without being pre-trained on specially curated multilingual parallel corpora. It remains a challenging problem to explain the underlying mechanisms by which LLMs process multilingual texts. In this paper, we delve into the composition of Transformer architectures in LLMs to pinpoint language-specific regions. Specially, we propose a novel detection method, language activation probability entropy (LAPE), to identify language-specific neurons within LLMs. Based on LAPE, we conduct comprehensive experiments on several representative LLMs, such as LLaMA-2, BLOOM, and Mistral. Our findings indicate that LLMs' proficiency in processing a particular language is predominantly due to a small subset of neurons, primarily situated in the models' top and bottom layers. Furthermore, we showcase the feasibility to "steer" the output language of LLMs by selectively activating or deactivating language-specific neurons. Our research provides important evidence to the understanding and exploration of the multilingual capabilities of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_16438 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models Tang, Tianyi Luo, Wenyang Huang, Haoyang Zhang, Dongdong Wang, Xiaolei Zhao, Xin Wei, Furu Wen, Ji-Rong Computation and Language Large language models (LLMs) demonstrate remarkable multilingual capabilities without being pre-trained on specially curated multilingual parallel corpora. It remains a challenging problem to explain the underlying mechanisms by which LLMs process multilingual texts. In this paper, we delve into the composition of Transformer architectures in LLMs to pinpoint language-specific regions. Specially, we propose a novel detection method, language activation probability entropy (LAPE), to identify language-specific neurons within LLMs. Based on LAPE, we conduct comprehensive experiments on several representative LLMs, such as LLaMA-2, BLOOM, and Mistral. Our findings indicate that LLMs' proficiency in processing a particular language is predominantly due to a small subset of neurons, primarily situated in the models' top and bottom layers. Furthermore, we showcase the feasibility to "steer" the output language of LLMs by selectively activating or deactivating language-specific neurons. Our research provides important evidence to the understanding and exploration of the multilingual capabilities of LLMs. |
| title | Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.16438 |