Optimizer-Induced Mode Connectivity: From AdamW to Muon
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914551896735744 |
|---|---|
| author | Zhang, Fangzhao Kim, Sungyoon Zhang, Erica Jiang, Yiqi Pilanci, Mert |
| author_facet | Zhang, Fangzhao Kim, Sungyoon Zhang, Erica Jiang, Yiqi Pilanci, Mert |
| contents | Mode connectivity has been widely studied, yet the role of the optimizer remains underexplored. We revisit it through optimizer-induced implicit regularization, asking how connectivity behaves when restricted to solutions constrained by a given optimizer. For two-layer ReLU networks, we show that solutions from a single optimizer -- AdamW, Muon, or others in the Lion-$\mathcal{K}$ family -- form a connected set at sufficiently large width, a result not implied by prior work. We then characterize how optimizer-induced regions interact: at large width two different regions can be disjoint or overlap depending on regularization, while in our small-width example AdamW and Muon converge to disconnected zero-loss components separated by a provable loss barrier. Empirically, in GPT-2 pretraining, we observe same-optimizer paths preserve each model's spectrum while cross-optimizer paths traverse a smooth transition. Our results reveal optimizer-dependent structure beyond classical mode connectivity literature. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_09991 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Optimizer-Induced Mode Connectivity: From AdamW to Muon Zhang, Fangzhao Kim, Sungyoon Zhang, Erica Jiang, Yiqi Pilanci, Mert Artificial Intelligence Machine Learning Optimization and Control Mode connectivity has been widely studied, yet the role of the optimizer remains underexplored. We revisit it through optimizer-induced implicit regularization, asking how connectivity behaves when restricted to solutions constrained by a given optimizer. For two-layer ReLU networks, we show that solutions from a single optimizer -- AdamW, Muon, or others in the Lion-$\mathcal{K}$ family -- form a connected set at sufficiently large width, a result not implied by prior work. We then characterize how optimizer-induced regions interact: at large width two different regions can be disjoint or overlap depending on regularization, while in our small-width example AdamW and Muon converge to disconnected zero-loss components separated by a provable loss barrier. Empirically, in GPT-2 pretraining, we observe same-optimizer paths preserve each model's spectrum while cross-optimizer paths traverse a smooth transition. Our results reveal optimizer-dependent structure beyond classical mode connectivity literature. |
| title | Optimizer-Induced Mode Connectivity: From AdamW to Muon |
| topic | Artificial Intelligence Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2605.09991 |