Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915515965898752 |
|---|---|
| author | Deng, Guanzhi Liu, Mingyang Wu, Dapeng Li, Yinqiao Song, Linqi |
| author_facet | Deng, Guanzhi Liu, Mingyang Wu, Dapeng Li, Yinqiao Song, Linqi |
| contents | Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning method for large language models. However, its linear nature limits expressiveness. We propose LoRAN, a non-linear extension of LoRA that applies lightweight transformations to the low-rank updates. We further introduce Sinter, a sine-based activation that adds structured perturbations without increasing parameter count. Experiments across summarization and classification tasks show that LoRAN consistently improves over QLoRA. Ablation studies reveal that Sinter outperforms standard activations such as Sigmoid, ReLU, and Tanh, highlighting the importance of activation design in lowrank tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_21870 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations Deng, Guanzhi Liu, Mingyang Wu, Dapeng Li, Yinqiao Song, Linqi Computation and Language Artificial Intelligence Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning method for large language models. However, its linear nature limits expressiveness. We propose LoRAN, a non-linear extension of LoRA that applies lightweight transformations to the low-rank updates. We further introduce Sinter, a sine-based activation that adds structured perturbations without increasing parameter count. Experiments across summarization and classification tasks show that LoRAN consistently improves over QLoRA. Ablation studies reveal that Sinter outperforms standard activations such as Sigmoid, ReLU, and Tanh, highlighting the importance of activation design in lowrank tuning. |
| title | Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.21870 |