Training Neural Networks from Scratch with Parallel Low-Rank Adapters
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929438813323264 |
|---|---|
| author | Huh, Minyoung Cheung, Brian Bernstein, Jeremy Isola, Phillip Agrawal, Pulkit |
| author_facet | Huh, Minyoung Cheung, Brian Bernstein, Jeremy Isola, Phillip Agrawal, Pulkit |
| contents | The scalability of deep learning models is fundamentally limited by computing resources, memory, and communication. Although methods like low-rank adaptation (LoRA) have reduced the cost of model finetuning, its application in model pre-training remains largely unexplored. This paper explores extending LoRA to model pre-training, identifying the inherent constraints and limitations of standard LoRA in this context. We introduce LoRA-the-Explorer (LTE), a novel bi-level optimization algorithm designed to enable parallel training of multiple low-rank heads across computing nodes, thereby reducing the need for frequent synchronization. Our approach includes extensive experimentation on vision transformers using various vision datasets, demonstrating that LTE is competitive with standard pre-training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_16828 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Training Neural Networks from Scratch with Parallel Low-Rank Adapters Huh, Minyoung Cheung, Brian Bernstein, Jeremy Isola, Phillip Agrawal, Pulkit Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition The scalability of deep learning models is fundamentally limited by computing resources, memory, and communication. Although methods like low-rank adaptation (LoRA) have reduced the cost of model finetuning, its application in model pre-training remains largely unexplored. This paper explores extending LoRA to model pre-training, identifying the inherent constraints and limitations of standard LoRA in this context. We introduce LoRA-the-Explorer (LTE), a novel bi-level optimization algorithm designed to enable parallel training of multiple low-rank heads across computing nodes, thereby reducing the need for frequent synchronization. Our approach includes extensive experimentation on vision transformers using various vision datasets, demonstrating that LTE is competitive with standard pre-training. |
| title | Training Neural Networks from Scratch with Parallel Low-Rank Adapters |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.16828 |