LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917941925117952 |
|---|---|
| author | Robert, Thomas Safaryan, Mher Modoranu, Ionut-Vlad Alistarh, Dan |
| author_facet | Robert, Thomas Safaryan, Mher Modoranu, Ionut-Vlad Alistarh, Dan |
| contents | We introduce LDAdam, a memory-efficient optimizer for training large models, that performs adaptive optimization steps within lower dimensional subspaces, while consistently exploring the full parameter space during training. This strategy keeps the optimizer's memory footprint to a fraction of the model size. LDAdam relies on a new projection-aware update rule for the optimizer states that allows for transitioning between subspaces, i.e., estimation of the statistics of the projected gradients. To mitigate the errors due to low-rank projection, LDAdam integrates a new generalized error feedback mechanism, which explicitly accounts for both gradient and optimizer state compression. We prove the convergence of LDAdam under standard assumptions, and show that LDAdam allows for accurate and efficient fine-tuning and pre-training of language models. Code is available at https://github.com/IST-DASLab/LDAdam |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_16103 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics Robert, Thomas Safaryan, Mher Modoranu, Ionut-Vlad Alistarh, Dan Machine Learning Optimization and Control We introduce LDAdam, a memory-efficient optimizer for training large models, that performs adaptive optimization steps within lower dimensional subspaces, while consistently exploring the full parameter space during training. This strategy keeps the optimizer's memory footprint to a fraction of the model size. LDAdam relies on a new projection-aware update rule for the optimizer states that allows for transitioning between subspaces, i.e., estimation of the statistics of the projected gradients. To mitigate the errors due to low-rank projection, LDAdam integrates a new generalized error feedback mechanism, which explicitly accounts for both gradient and optimizer state compression. We prove the convergence of LDAdam under standard assumptions, and show that LDAdam allows for accurate and efficient fine-tuning and pre-training of language models. Code is available at https://github.com/IST-DASLab/LDAdam |
| title | LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2410.16103 |