Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.12402 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911463654817792 |
|---|---|
| author | Chen, Lizhang Li, Jonathan Liang, Kaizhao Su, Baiyu Xie, Cong Pierse, Nuo Wang Liang, Chen Lao, Ni Liu, Qiang |
| author_facet | Chen, Lizhang Li, Jonathan Liang, Kaizhao Su, Baiyu Xie, Cong Pierse, Nuo Wang Liang, Chen Lao, Ni Liu, Qiang |
| contents | We introduce Cautious Weight Decay (CWD), a one-line, optimizer-agnostic modification that applies weight decay only to parameter coordinates whose signs align with the optimizer update. Unlike standard decoupled decay, which implicitly optimizes a regularized or constrained objective, CWD preserves the original loss and admits a bilevel interpretation: it induces sliding-mode behavior upon reaching the stationary manifold, allowing it to search for locally Pareto-optimal stationary points of the unmodified objective. In practice, CWD is a drop-in change for optimizers such as AdamW, Lion, and Muon, requiring no new hyperparameters or additional tuning. For language model pre-training and ImageNet classification, CWD consistently improves final loss and accuracy at million- to billion-parameter scales. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_12402 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Cautious Weight Decay Chen, Lizhang Li, Jonathan Liang, Kaizhao Su, Baiyu Xie, Cong Pierse, Nuo Wang Liang, Chen Lao, Ni Liu, Qiang Machine Learning Optimization and Control We introduce Cautious Weight Decay (CWD), a one-line, optimizer-agnostic modification that applies weight decay only to parameter coordinates whose signs align with the optimizer update. Unlike standard decoupled decay, which implicitly optimizes a regularized or constrained objective, CWD preserves the original loss and admits a bilevel interpretation: it induces sliding-mode behavior upon reaching the stationary manifold, allowing it to search for locally Pareto-optimal stationary points of the unmodified objective. In practice, CWD is a drop-in change for optimizers such as AdamW, Lion, and Muon, requiring no new hyperparameters or additional tuning. For language model pre-training and ImageNet classification, CWD consistently improves final loss and accuracy at million- to billion-parameter scales. |
| title | Cautious Weight Decay |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2510.12402 |