Model Merging in Pre-training of Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913851855863808 |
|---|---|
| author | Li, Yunshui Ma, Yiyuan Yan, Shen Zhang, Chaoyi Liu, Jing Lu, Jianqiao Xu, Ziwen Chen, Mengzhao Wang, Minrui Zhan, Shiyi Ma, Jin Lai, Xunhao Liu, Deyi Luo, Yao Bin, Xingyan Ren, Hongbin Han, Mingji Hao, Wenhao Yi, Bairen Liu, LingJun Ma, Bole Jia, Xiaoying Zhou, Xun Qiao, Siyuan Xiang, Liang Wu, Yonghui |
| author_facet | Li, Yunshui Ma, Yiyuan Yan, Shen Zhang, Chaoyi Liu, Jing Lu, Jianqiao Xu, Ziwen Chen, Mengzhao Wang, Minrui Zhan, Shiyi Ma, Jin Lai, Xunhao Liu, Deyi Luo, Yao Bin, Xingyan Ren, Hongbin Han, Mingji Hao, Wenhao Yi, Bairen Liu, LingJun Ma, Bole Jia, Xiaoying Zhou, Xun Qiao, Siyuan Xiang, Liang Wu, Yonghui |
| contents | Model merging has emerged as a promising technique for enhancing large language models, though its application in large-scale pre-training remains relatively unexplored. In this paper, we present a comprehensive investigation of model merging techniques during the pre-training process. Through extensive experiments with both dense and Mixture-of-Experts (MoE) architectures ranging from millions to over 100 billion parameters, we demonstrate that merging checkpoints trained with constant learning rates not only achieves significant performance improvements but also enables accurate prediction of annealing behavior. These improvements lead to both more efficient model development and significantly lower training costs. Our detailed ablation studies on merging strategies and hyperparameters provide new insights into the underlying mechanisms while uncovering novel applications. Through comprehensive experimental analysis, we offer the open-source community practical pre-training guidelines for effective model merging. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_12082 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Model Merging in Pre-training of Large Language Models Li, Yunshui Ma, Yiyuan Yan, Shen Zhang, Chaoyi Liu, Jing Lu, Jianqiao Xu, Ziwen Chen, Mengzhao Wang, Minrui Zhan, Shiyi Ma, Jin Lai, Xunhao Liu, Deyi Luo, Yao Bin, Xingyan Ren, Hongbin Han, Mingji Hao, Wenhao Yi, Bairen Liu, LingJun Ma, Bole Jia, Xiaoying Zhou, Xun Qiao, Siyuan Xiang, Liang Wu, Yonghui Computation and Language Machine Learning Model merging has emerged as a promising technique for enhancing large language models, though its application in large-scale pre-training remains relatively unexplored. In this paper, we present a comprehensive investigation of model merging techniques during the pre-training process. Through extensive experiments with both dense and Mixture-of-Experts (MoE) architectures ranging from millions to over 100 billion parameters, we demonstrate that merging checkpoints trained with constant learning rates not only achieves significant performance improvements but also enables accurate prediction of annealing behavior. These improvements lead to both more efficient model development and significantly lower training costs. Our detailed ablation studies on merging strategies and hyperparameters provide new insights into the underlying mechanisms while uncovering novel applications. Through comprehensive experimental analysis, we offer the open-source community practical pre-training guidelines for effective model merging. |
| title | Model Merging in Pre-training of Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2505.12082 |