SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910775600218112 |
|---|---|
| author | Fan, Yuchun Mu, Yongyu Wang, Yilin Huang, Lei Ruan, Junhao Li, Bei Xiao, Tong Huang, Shujian Feng, Xiaocheng Zhu, Jingbo |
| author_facet | Fan, Yuchun Mu, Yongyu Wang, Yilin Huang, Lei Ruan, Junhao Li, Bei Xiao, Tong Huang, Shujian Feng, Xiaocheng Zhu, Jingbo |
| contents | Despite the significant improvements achieved by large language models (LLMs) in English reasoning tasks, these models continue to struggle with multilingual reasoning. Recent studies leverage a full-parameter and two-stage training paradigm to teach models to first understand non-English questions and then reason. However, this method suffers from both substantial computational resource computing and catastrophic forgetting. The fundamental cause is that, with the primary goal of enhancing multilingual comprehension, an excessive number of irrelevant layers and parameters are tuned during the first stage. Given our findings that the representation learning of languages is merely conducted in lower-level layers, we propose an efficient multilingual reasoning alignment approach that precisely identifies and fine-tunes the layers responsible for handling multilingualism. Experimental results show that our method, SLAM, only tunes 6 layers' feed-forward sub-layers including 6.5-8% of all parameters within 7B and 13B LLMs, achieving superior average performance than all strong baselines across 10 languages. Meanwhile, SLAM only involves one training stage, reducing training time by 4.1-11.9 compared to the two-stage method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_03681 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment Fan, Yuchun Mu, Yongyu Wang, Yilin Huang, Lei Ruan, Junhao Li, Bei Xiao, Tong Huang, Shujian Feng, Xiaocheng Zhu, Jingbo Computation and Language Artificial Intelligence Despite the significant improvements achieved by large language models (LLMs) in English reasoning tasks, these models continue to struggle with multilingual reasoning. Recent studies leverage a full-parameter and two-stage training paradigm to teach models to first understand non-English questions and then reason. However, this method suffers from both substantial computational resource computing and catastrophic forgetting. The fundamental cause is that, with the primary goal of enhancing multilingual comprehension, an excessive number of irrelevant layers and parameters are tuned during the first stage. Given our findings that the representation learning of languages is merely conducted in lower-level layers, we propose an efficient multilingual reasoning alignment approach that precisely identifies and fine-tunes the layers responsible for handling multilingualism. Experimental results show that our method, SLAM, only tunes 6 layers' feed-forward sub-layers including 6.5-8% of all parameters within 7B and 13B LLMs, achieving superior average performance than all strong baselines across 10 languages. Meanwhile, SLAM only involves one training stage, reducing training time by 4.1-11.9 compared to the two-stage method. |
| title | SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2501.03681 |