Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.13498 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918377552871424 |
|---|---|
| author | Cheng, Peng Zang, Jiucheng Li, Qingnan Ma, Liheng Cui, Yufei Zhang, Yingxue Chen, Boxing Jian, Ming Tong, Wen |
| author_facet | Cheng, Peng Zang, Jiucheng Li, Qingnan Ma, Liheng Cui, Yufei Zhang, Yingxue Chen, Boxing Jian, Ming Tong, Wen |
| contents | Muon-style optimizers leverage Newton-Schulz (NS) iterations to orthogonalize updates, yielding update geometries that often outperform Adam-series methods. However, this orthogonalization discards magnitude information, rendering training sensitive to step-size hyperparameters and vulnerable to high-energy bursts. To mitigate this, we introduce TrasMuon (\textbf{T}rust \textbf{R}egion \textbf{A}daptive \textbf{S}caling \textbf{Muon}). TrasMuon preserves the near-isometric geometry of Muon while stabilizing magnitudes through (i) global RMS calibration and (ii) energy-based trust-region clipping. We demonstrate that while reintroducing adaptive scaling improves optimization efficiency, it typically exacerbates instability due to high-energy outliers. TrasMuon addresses this by defining a trust region based on relative energy ratios, confining updates to a stable zone. Empirical experiments on vision and language models demonstrate that TrasMuon converges faster than baselines. Furthermore, experiments without warmup stages confirm TrasMuon's superior stability and robustness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_13498 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizers Cheng, Peng Zang, Jiucheng Li, Qingnan Ma, Liheng Cui, Yufei Zhang, Yingxue Chen, Boxing Jian, Ming Tong, Wen Machine Learning Artificial Intelligence Muon-style optimizers leverage Newton-Schulz (NS) iterations to orthogonalize updates, yielding update geometries that often outperform Adam-series methods. However, this orthogonalization discards magnitude information, rendering training sensitive to step-size hyperparameters and vulnerable to high-energy bursts. To mitigate this, we introduce TrasMuon (\textbf{T}rust \textbf{R}egion \textbf{A}daptive \textbf{S}caling \textbf{Muon}). TrasMuon preserves the near-isometric geometry of Muon while stabilizing magnitudes through (i) global RMS calibration and (ii) energy-based trust-region clipping. We demonstrate that while reintroducing adaptive scaling improves optimization efficiency, it typically exacerbates instability due to high-energy outliers. TrasMuon addresses this by defining a trust region based on relative energy ratios, confining updates to a stable zone. Empirical experiments on vision and language models demonstrate that TrasMuon converges faster than baselines. Furthermore, experiments without warmup stages confirm TrasMuon's superior stability and robustness. |
| title | TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizers |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2602.13498 |