WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912532232404992 |
|---|---|
| author | Tian, Changxin Wang, Jiapeng Zhao, Qian Chen, Kunlong Liu, Jia Liu, Ziqi Mao, Jiaxin Zhao, Wayne Xin Zhang, Zhiqiang Zhou, Jun |
| author_facet | Tian, Changxin Wang, Jiapeng Zhao, Qian Chen, Kunlong Liu, Jia Liu, Ziqi Mao, Jiaxin Zhao, Wayne Xin Zhang, Zhiqiang Zhou, Jun |
| contents | Recent advances in learning rate (LR) scheduling have demonstrated the effectiveness of decay-free approaches that eliminate the traditional decay phase while maintaining competitive performance. Model merging techniques have emerged as particularly promising solutions in this domain. We present Warmup-Stable and Merge (WSM), a general framework that establishes a formal connection between learning rate decay and model merging. WSM provides a unified theoretical foundation for emulating various decay strategies-including cosine decay, linear decay and inverse square root decay-as principled model averaging schemes, while remaining fully compatible with diverse optimization methods. Through extensive experiments, we identify merge duration-the training window for checkpoint aggregation-as the most critical factor influencing model performance, surpassing the importance of both checkpoint interval and merge quantity. Our framework consistently outperforms the widely-adopted Warmup-Stable-Decay (WSD) approach across multiple benchmarks, achieving significant improvements of +3.5% on MATH, +2.9% on HumanEval, and +5.5% on MMLU-Pro. The performance advantages extend to supervised fine-tuning scenarios, highlighting WSM's potential for long-term model refinement. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_17634 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training Tian, Changxin Wang, Jiapeng Zhao, Qian Chen, Kunlong Liu, Jia Liu, Ziqi Mao, Jiaxin Zhao, Wayne Xin Zhang, Zhiqiang Zhou, Jun Computation and Language Machine Learning I.2.7 Recent advances in learning rate (LR) scheduling have demonstrated the effectiveness of decay-free approaches that eliminate the traditional decay phase while maintaining competitive performance. Model merging techniques have emerged as particularly promising solutions in this domain. We present Warmup-Stable and Merge (WSM), a general framework that establishes a formal connection between learning rate decay and model merging. WSM provides a unified theoretical foundation for emulating various decay strategies-including cosine decay, linear decay and inverse square root decay-as principled model averaging schemes, while remaining fully compatible with diverse optimization methods. Through extensive experiments, we identify merge duration-the training window for checkpoint aggregation-as the most critical factor influencing model performance, surpassing the importance of both checkpoint interval and merge quantity. Our framework consistently outperforms the widely-adopted Warmup-Stable-Decay (WSD) approach across multiple benchmarks, achieving significant improvements of +3.5% on MATH, +2.9% on HumanEval, and +5.5% on MMLU-Pro. The performance advantages extend to supervised fine-tuning scenarios, highlighting WSM's potential for long-term model refinement. |
| title | WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training |
| topic | Computation and Language Machine Learning I.2.7 |
| url | https://arxiv.org/abs/2507.17634 |