LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913687859625984 |
|---|---|
| author | Zhou, Zikai Zhang, Qizheng Kumbong, Hermann Olukotun, Kunle |
| author_facet | Zhou, Zikai Zhang, Qizheng Kumbong, Hermann Olukotun, Kunle |
| contents | Fine-tuning large language models (LLMs) is increasingly costly as models scale to hundreds of billions of parameters, and even parameter-efficient fine-tuning (PEFT) methods like LoRA remain resource-intensive. We introduce LowRA, the first framework to enable LoRA fine-tuning below 2 bits per parameter with minimal performance loss. LowRA optimizes fine-grained quantization - mapping, threshold selection, and precision assignment - while leveraging efficient CUDA kernels for scalable deployment. Extensive evaluations across 4 LLMs and 4 datasets show that LowRA achieves a superior performance-precision trade-off above 2 bits and remains accurate down to 1.15 bits, reducing memory usage by up to 50%. Our results highlight the potential of ultra-low-bit LoRA fine-tuning for resource-constrained environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_08141 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits Zhou, Zikai Zhang, Qizheng Kumbong, Hermann Olukotun, Kunle Machine Learning Hardware Architecture Computation and Language Performance Fine-tuning large language models (LLMs) is increasingly costly as models scale to hundreds of billions of parameters, and even parameter-efficient fine-tuning (PEFT) methods like LoRA remain resource-intensive. We introduce LowRA, the first framework to enable LoRA fine-tuning below 2 bits per parameter with minimal performance loss. LowRA optimizes fine-grained quantization - mapping, threshold selection, and precision assignment - while leveraging efficient CUDA kernels for scalable deployment. Extensive evaluations across 4 LLMs and 4 datasets show that LowRA achieves a superior performance-precision trade-off above 2 bits and remains accurate down to 1.15 bits, reducing memory usage by up to 50%. Our results highlight the potential of ultra-low-bit LoRA fine-tuning for resource-constrained environments. |
| title | LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits |
| topic | Machine Learning Hardware Architecture Computation and Language Performance |
| url | https://arxiv.org/abs/2502.08141 |