Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.13034 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915675485765632 |
|---|---|
| author | He, Xiaoyu Cai, Yu Jia, Jin Huang, Canxi Chen, Wenqing Zheng, Zibin |
| author_facet | He, Xiaoyu Cai, Yu Jia, Jin Huang, Canxi Chen, Wenqing Zheng, Zibin |
| contents | This work proposes Alada, an adaptive momentum method for stochastic optimization over large-scale matrices. Alada employs a rank-one factorization approach to estimate the second moment of gradients, where factors are updated alternatively to minimize the estimation error. Alada achieves sublinear memory overheads and can be readily extended to optimizing tensor-shaped variables.We also equip Alada with a first moment estimation rule, which enhances the algorithm's robustness without incurring additional memory overheads. The theoretical performance of Alada aligns with that of traditional methods such as Adam. Numerical studies conducted on several natural language processing tasks demonstrate the reduction in memory overheads and the robustness in training large models relative to Adam and its variants. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_13034 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Alada: Alternating Adaptation of Momentum Method for Memory-Efficient Matrix Optimization He, Xiaoyu Cai, Yu Jia, Jin Huang, Canxi Chen, Wenqing Zheng, Zibin Machine Learning This work proposes Alada, an adaptive momentum method for stochastic optimization over large-scale matrices. Alada employs a rank-one factorization approach to estimate the second moment of gradients, where factors are updated alternatively to minimize the estimation error. Alada achieves sublinear memory overheads and can be readily extended to optimizing tensor-shaped variables.We also equip Alada with a first moment estimation rule, which enhances the algorithm's robustness without incurring additional memory overheads. The theoretical performance of Alada aligns with that of traditional methods such as Adam. Numerical studies conducted on several natural language processing tasks demonstrate the reduction in memory overheads and the robustness in training large models relative to Adam and its variants. |
| title | Alada: Alternating Adaptation of Momentum Method for Memory-Efficient Matrix Optimization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2512.13034 |