Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.13811 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915435871469568 |
|---|---|
| author | Torroba-Hennigen, Lucas Lang, Hunter Guo, Han Kim, Yoon |
| author_facet | Torroba-Hennigen, Lucas Lang, Hunter Guo, Han Kim, Yoon |
| contents | We study memory-efficient optimization of neural networks (in particular language models) with linear gradient transformations, where the gradients are linearly mapped to a lower dimensional space than the full parameter space, thus saving memory required for gradient accumulation and optimizer state persistence. The model parameters are updated by first performing an optimization step in the lower dimensional space and then going back into the original parameter space via the linear map's transpose. We show that optimizing the model in this transformed space is equivalent to reparameterizing the original model through a linear adapter that additively modifies the model parameters, and then only optimizing the adapter's parameters. When the transformation is Kronecker-factored, this establishes an equivalence between GaLore and one-sided LoRA. We show that this duality between gradient transformations and adapter-based reparameterizations unifies existing approaches to memory-efficient training and suggests new techniques for improving training efficiency and memory use. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13811 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Duality between Gradient Transformations and Adapters Torroba-Hennigen, Lucas Lang, Hunter Guo, Han Kim, Yoon Machine Learning Computation and Language We study memory-efficient optimization of neural networks (in particular language models) with linear gradient transformations, where the gradients are linearly mapped to a lower dimensional space than the full parameter space, thus saving memory required for gradient accumulation and optimizer state persistence. The model parameters are updated by first performing an optimization step in the lower dimensional space and then going back into the original parameter space via the linear map's transpose. We show that optimizing the model in this transformed space is equivalent to reparameterizing the original model through a linear adapter that additively modifies the model parameters, and then only optimizing the adapter's parameters. When the transformation is Kronecker-factored, this establishes an equivalence between GaLore and one-sided LoRA. We show that this duality between gradient transformations and adapter-based reparameterizations unifies existing approaches to memory-efficient training and suggests new techniques for improving training efficiency and memory use. |
| title | On the Duality between Gradient Transformations and Adapters |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2502.13811 |