Why LoRA Resists Label Noise: A Theoretical Framework for Noise-Robust Parameter-Efficient Fine-Tuning
Fuente:
arXiv
Enregistré dans:
| Auteur principal: | |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918316379996160 |
|---|---|
| author | Steele, Brady |
| author_facet | Steele, Brady |
| contents | Parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) have become the dominant paradigm for adapting large pretrained models. We present a theoretical framework explaining an underexplored property: LoRA's inherent resistance to label noise. Our analysis reveals three key insights. First, we prove that rank-$r$ LoRA cannot memorize all possible label assignments once the sample size exceeds $O(r(d+k-r))$, limiting its capacity to fit arbitrary noise. Second, we derive an optimal rank balancing approximation bias and noise-induced variance, showing it decreases with noise rate. Third, we establish temporal separation: clean patterns are learned early while noise memorization occurs later. We propose RACT (Rank-Aware Curriculum Training), leveraging rank discrepancy for noise detection. Experiments validate our predictions, with RACT achieving 91.1% F1 for noise detection on AG News while maintaining 91.46% accuracy, competitive with baselines that lack noise detection capability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_00084 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Why LoRA Resists Label Noise: A Theoretical Framework for Noise-Robust Parameter-Efficient Fine-Tuning Steele, Brady Machine Learning I.2.6; I.2.7 Parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) have become the dominant paradigm for adapting large pretrained models. We present a theoretical framework explaining an underexplored property: LoRA's inherent resistance to label noise. Our analysis reveals three key insights. First, we prove that rank-$r$ LoRA cannot memorize all possible label assignments once the sample size exceeds $O(r(d+k-r))$, limiting its capacity to fit arbitrary noise. Second, we derive an optimal rank balancing approximation bias and noise-induced variance, showing it decreases with noise rate. Third, we establish temporal separation: clean patterns are learned early while noise memorization occurs later. We propose RACT (Rank-Aware Curriculum Training), leveraging rank discrepancy for noise detection. Experiments validate our predictions, with RACT achieving 91.1% F1 for noise detection on AG News while maintaining 91.46% accuracy, competitive with baselines that lack noise detection capability. |
| title | Why LoRA Resists Label Noise: A Theoretical Framework for Noise-Robust Parameter-Efficient Fine-Tuning |
| topic | Machine Learning I.2.6; I.2.7 |
| url | https://arxiv.org/abs/2602.00084 |