SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915768957927424 |
|---|---|
| author | Yu, Qifan Ma, Xinyu Zhuo, Zhijian Wang, Minrui Liu, Deyi Zhan, Shiyi Ma, Yiyuan Xiang, Liang Bin, Xingyan He, Di |
| author_facet | Yu, Qifan Ma, Xinyu Zhuo, Zhijian Wang, Minrui Liu, Deyi Zhan, Shiyi Ma, Yiyuan Xiang, Liang Bin, Xingyan He, Di |
| contents | Progressive Learning (PL) reduces pre-training computational overhead by gradually increasing model scale. While prior work has extensively explored depth expansion, width expansion remains significantly understudied, with the few existing methods limited to the early stages of training. However, expanding width during the mid-stage is essential for maximizing computational savings, yet it remains a formidable challenge due to severe training instabilities. Empirically, we show that naive initialization at this stage disrupts activation statistics, triggering loss spikes, while copy-based initialization introduces gradient symmetry that hinders feature diversity. To address these issues, we propose SPARKLING (balancing {S}ignal {P}reservation {A}nd symmet{R}y brea{K}ing for width-progressive {L}earn{ING}), a novel framework for mid-stage width expansion. Our method achieves signal preservation via RMS-scale consistency, stabilizing activation statistics during expansion. Symmetry breaking is ensured through asymmetric optimizer state resetting and learning rate re-warmup. Extensive experiments on Mixture-of-Experts (MoE) models demonstrate that, across multiple width axes and optimizer families, SPARKLING consistently outperforms training from scratch and reduces training cost by up to 35% under $2\times$ width expansion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_02472 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning Yu, Qifan Ma, Xinyu Zhuo, Zhijian Wang, Minrui Liu, Deyi Zhan, Shiyi Ma, Yiyuan Xiang, Liang Bin, Xingyan He, Di Machine Learning Computation and Language Progressive Learning (PL) reduces pre-training computational overhead by gradually increasing model scale. While prior work has extensively explored depth expansion, width expansion remains significantly understudied, with the few existing methods limited to the early stages of training. However, expanding width during the mid-stage is essential for maximizing computational savings, yet it remains a formidable challenge due to severe training instabilities. Empirically, we show that naive initialization at this stage disrupts activation statistics, triggering loss spikes, while copy-based initialization introduces gradient symmetry that hinders feature diversity. To address these issues, we propose SPARKLING (balancing {S}ignal {P}reservation {A}nd symmet{R}y brea{K}ing for width-progressive {L}earn{ING}), a novel framework for mid-stage width expansion. Our method achieves signal preservation via RMS-scale consistency, stabilizing activation statistics during expansion. Symmetry breaking is ensured through asymmetric optimizer state resetting and learning rate re-warmup. Extensive experiments on Mixture-of-Experts (MoE) models demonstrate that, across multiple width axes and optimizer families, SPARKLING consistently outperforms training from scratch and reduces training cost by up to 35% under $2\times$ width expansion. |
| title | SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2602.02472 |