Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.31371 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910272399081472 |
|---|---|
| author | Feoktistov, Dmitrii Belinsky, Timofey Veprikov, Andrey Zainullin, Amir Beznosikov, Aleksandr |
| author_facet | Feoktistov, Dmitrii Belinsky, Timofey Veprikov, Andrey Zainullin, Amir Beznosikov, Aleksandr |
| contents | Sign-based and LMO-inspired optimizers have recently attracted substantial attention in deep learning due to their strong performance and low memory footprint. However, their fixed-magnitude updates can hurt terminal convergence: they decouple update mechanisms from gradient magnitudes and fail to account for parameter heterogeneity, often leading to oscillation rather than convergence. We propose SoftSignum, a smooth relaxation of sign-based optimization that replaces the hard sign map with a temperature-controlled soft-sign transformation, enabling a parameter-wise transition from sign-like updates to magnitude-sensitive SGD-like steps. We complement it with an adaptive quantile-based temperature schedule and extend the same principle to matrix-valued optimizers, obtaining SoftMuon. We also develop a generalized geometry-relaxation framework based on strongly convex regularizers and Fenchel conjugates, proving convergence in stochastic non-convex setting. Experiments on diverse deep learning tasks, including LLM pretraining, show that SoftSignum and SoftMuon consistently improve over their hard sign-based counterparts and standard AdamW. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_31371 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling Feoktistov, Dmitrii Belinsky, Timofey Veprikov, Andrey Zainullin, Amir Beznosikov, Aleksandr Machine Learning Sign-based and LMO-inspired optimizers have recently attracted substantial attention in deep learning due to their strong performance and low memory footprint. However, their fixed-magnitude updates can hurt terminal convergence: they decouple update mechanisms from gradient magnitudes and fail to account for parameter heterogeneity, often leading to oscillation rather than convergence. We propose SoftSignum, a smooth relaxation of sign-based optimization that replaces the hard sign map with a temperature-controlled soft-sign transformation, enabling a parameter-wise transition from sign-like updates to magnitude-sensitive SGD-like steps. We complement it with an adaptive quantile-based temperature schedule and extend the same principle to matrix-valued optimizers, obtaining SoftMuon. We also develop a generalized geometry-relaxation framework based on strongly convex regularizers and Fenchel conjugates, proving convergence in stochastic non-convex setting. Experiments on diverse deep learning tasks, including LLM pretraining, show that SoftSignum and SoftMuon consistently improve over their hard sign-based counterparts and standard AdamW. |
| title | Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.31371 |