Low-Rank Adaptation for Foundation Models: A Comprehensive Review
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909883655258112 |
|---|---|
| author | Yang, Menglin Chen, Jialin Tao, Jinkai Zhang, Yifei Liu, Jiahong Zhang, Jiasheng Ma, Qiyao Verma, Harshit Zhang, Regina Zhou, Min King, Irwin Ying, Rex |
| author_facet | Yang, Menglin Chen, Jialin Tao, Jinkai Zhang, Yifei Liu, Jiahong Zhang, Jiasheng Ma, Qiyao Verma, Harshit Zhang, Regina Zhou, Min King, Irwin Ying, Rex |
| contents | The rapid advancement of foundation modelslarge-scale neural networks trained on diverse, extensive datasetshas revolutionized artificial intelligence, enabling unprecedented advancements across domains such as natural language processing, computer vision, and scientific discovery. However, the substantial parameter count of these models, often reaching billions or trillions, poses significant challenges in adapting them to specific downstream tasks. Low-Rank Adaptation (LoRA) has emerged as a highly promising approach for mitigating these challenges, offering a parameter-efficient mechanism to fine-tune foundation models with minimal computational overhead. This survey provides the first comprehensive review of LoRA techniques beyond large Language Models to general foundation models, including recent techniques foundations, emerging frontiers and applications of low-rank adaptation across multiple domains. Finally, this survey discusses key challenges and future research directions in theoretical understanding, scalability, and robustness. This survey serves as a valuable resource for researchers and practitioners working with efficient foundation model adaptation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_00365 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Low-Rank Adaptation for Foundation Models: A Comprehensive Review Yang, Menglin Chen, Jialin Tao, Jinkai Zhang, Yifei Liu, Jiahong Zhang, Jiasheng Ma, Qiyao Verma, Harshit Zhang, Regina Zhou, Min King, Irwin Ying, Rex Machine Learning Artificial Intelligence I.2 The rapid advancement of foundation modelslarge-scale neural networks trained on diverse, extensive datasetshas revolutionized artificial intelligence, enabling unprecedented advancements across domains such as natural language processing, computer vision, and scientific discovery. However, the substantial parameter count of these models, often reaching billions or trillions, poses significant challenges in adapting them to specific downstream tasks. Low-Rank Adaptation (LoRA) has emerged as a highly promising approach for mitigating these challenges, offering a parameter-efficient mechanism to fine-tune foundation models with minimal computational overhead. This survey provides the first comprehensive review of LoRA techniques beyond large Language Models to general foundation models, including recent techniques foundations, emerging frontiers and applications of low-rank adaptation across multiple domains. Finally, this survey discusses key challenges and future research directions in theoretical understanding, scalability, and robustness. This survey serves as a valuable resource for researchers and practitioners working with efficient foundation model adaptation. |
| title | Low-Rank Adaptation for Foundation Models: A Comprehensive Review |
| topic | Machine Learning Artificial Intelligence I.2 |
| url | https://arxiv.org/abs/2501.00365 |