Adapprox: Adaptive Approximation in Adam Optimization via Randomized Low-Rank Matrices
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913278046765056 |
|---|---|
| author | Zhao, Pengxiang Li, Ping Gu, Yingjie Zheng, Yi Kölker, Stephan Ludger Wang, Zhefeng Yuan, Xiaoming |
| author_facet | Zhao, Pengxiang Li, Ping Gu, Yingjie Zheng, Yi Kölker, Stephan Ludger Wang, Zhefeng Yuan, Xiaoming |
| contents | As deep learning models exponentially increase in size, optimizers such as Adam encounter significant memory consumption challenges due to the storage of first and second moment data. Current memory-efficient methods like Adafactor and CAME often compromise accuracy with their matrix factorization techniques. Addressing this, we introduce Adapprox, a novel approach that employs randomized low-rank matrix approximation for a more effective and accurate approximation of Adam's second moment. Adapprox features an adaptive rank selection mechanism, finely balancing accuracy and memory efficiency, and includes an optional cosine similarity guidance strategy to enhance stability and expedite convergence. In GPT-2 training and downstream tasks, Adapprox surpasses AdamW by achieving 34.5% to 49.9% and 33.8% to 49.9% memory savings for the 117M and 345M models, respectively, with the first moment enabled, and further increases these savings without the first moment. Besides, it enhances convergence speed and improves downstream task performance relative to its counterparts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_14958 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Adapprox: Adaptive Approximation in Adam Optimization via Randomized Low-Rank Matrices Zhao, Pengxiang Li, Ping Gu, Yingjie Zheng, Yi Kölker, Stephan Ludger Wang, Zhefeng Yuan, Xiaoming Machine Learning Computation and Language Optimization and Control As deep learning models exponentially increase in size, optimizers such as Adam encounter significant memory consumption challenges due to the storage of first and second moment data. Current memory-efficient methods like Adafactor and CAME often compromise accuracy with their matrix factorization techniques. Addressing this, we introduce Adapprox, a novel approach that employs randomized low-rank matrix approximation for a more effective and accurate approximation of Adam's second moment. Adapprox features an adaptive rank selection mechanism, finely balancing accuracy and memory efficiency, and includes an optional cosine similarity guidance strategy to enhance stability and expedite convergence. In GPT-2 training and downstream tasks, Adapprox surpasses AdamW by achieving 34.5% to 49.9% and 33.8% to 49.9% memory savings for the 117M and 345M models, respectively, with the first moment enabled, and further increases these savings without the first moment. Besides, it enhances convergence speed and improves downstream task performance relative to its counterparts. |
| title | Adapprox: Adaptive Approximation in Adam Optimization via Randomized Low-Rank Matrices |
| topic | Machine Learning Computation and Language Optimization and Control |
| url | https://arxiv.org/abs/2403.14958 |