Advantageous Parameter Expansion Training Makes Better Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910975186173952 |
|---|---|
| author | Gu, Naibin Chen, Yilong Zhang, Zhenyu Fu, Peng Lin, Zheng Wang, Shuohuan Sun, Yu Wu, Hua Wang, Weiping Wang, Haifeng |
| author_facet | Gu, Naibin Chen, Yilong Zhang, Zhenyu Fu, Peng Lin, Zheng Wang, Shuohuan Sun, Yu Wu, Hua Wang, Weiping Wang, Haifeng |
| contents | Although scaling up the number of trainable parameters in both pre-training and fine-tuning can effectively improve the performance of large language models, it also leads to increased computational overhead. When delving into the parameter difference, we find that a subset of parameters, termed advantageous parameters, plays a crucial role in determining model performance. Further analysis reveals that stronger models tend to possess more such parameters. In this paper, we propose Advantageous Parameter EXpansion Training (APEX), a method that progressively expands advantageous parameters into the space of disadvantageous ones, thereby increasing their proportion and enhancing training effectiveness. Further theoretical analysis from the perspective of matrix effective rank explains the performance gains of APEX. Extensive experiments on both instruction tuning and continued pre-training demonstrate that, in instruction tuning, APEX outperforms full-parameter tuning while using only 52% of the trainable parameters. In continued pre-training, APEX achieves the same perplexity level as conventional training with just 33% of the training data, and yields significant improvements on downstream tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_24241 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Advantageous Parameter Expansion Training Makes Better Large Language Models Gu, Naibin Chen, Yilong Zhang, Zhenyu Fu, Peng Lin, Zheng Wang, Shuohuan Sun, Yu Wu, Hua Wang, Weiping Wang, Haifeng Computation and Language Although scaling up the number of trainable parameters in both pre-training and fine-tuning can effectively improve the performance of large language models, it also leads to increased computational overhead. When delving into the parameter difference, we find that a subset of parameters, termed advantageous parameters, plays a crucial role in determining model performance. Further analysis reveals that stronger models tend to possess more such parameters. In this paper, we propose Advantageous Parameter EXpansion Training (APEX), a method that progressively expands advantageous parameters into the space of disadvantageous ones, thereby increasing their proportion and enhancing training effectiveness. Further theoretical analysis from the perspective of matrix effective rank explains the performance gains of APEX. Extensive experiments on both instruction tuning and continued pre-training demonstrate that, in instruction tuning, APEX outperforms full-parameter tuning while using only 52% of the trainable parameters. In continued pre-training, APEX achieves the same perplexity level as conventional training with just 33% of the training data, and yields significant improvements on downstream tasks. |
| title | Advantageous Parameter Expansion Training Makes Better Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.24241 |