TensLoRA: Tensor Alternatives for Low-Rank Adaptation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911423487016960 |
|---|---|
| author | Marmoret, Axel Bensaid, Reda Lys, Jonathan Gripon, Vincent Leduc-Primeau, François |
| author_facet | Marmoret, Axel Bensaid, Reda Lys, Jonathan Gripon, Vincent Leduc-Primeau, François |
| contents | Low-Rank Adaptation (LoRA) is widely used to efficiently adapt Transformers by adding trainable low-rank matrices to attention projections. While effective, these matrices are considered independent for each attention projection (Query, Key, and Value) and each layer. Recent extensions have considered joint, tensor-based adaptations, but only in limited forms and without a systematic framework. We introduce TensLoRA, a unified framework that aggregates LoRA updates into higher-order tensors and models a broad family of tensor-based low-rank adaptations. Our formulation generalizes existing tensor-based methods and enables mode-specific compression rates, allowing parameter budgets to be tailored according to the modality and task. Experiments on vision and language benchmarks reveal that the tensor construction directly impacts performance, sometimes better than standard LoRA under similar parameter counts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_19391 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TensLoRA: Tensor Alternatives for Low-Rank Adaptation Marmoret, Axel Bensaid, Reda Lys, Jonathan Gripon, Vincent Leduc-Primeau, François Machine Learning Artificial Intelligence 68T I.2.6; I.2.7; I.2.10 Low-Rank Adaptation (LoRA) is widely used to efficiently adapt Transformers by adding trainable low-rank matrices to attention projections. While effective, these matrices are considered independent for each attention projection (Query, Key, and Value) and each layer. Recent extensions have considered joint, tensor-based adaptations, but only in limited forms and without a systematic framework. We introduce TensLoRA, a unified framework that aggregates LoRA updates into higher-order tensors and models a broad family of tensor-based low-rank adaptations. Our formulation generalizes existing tensor-based methods and enables mode-specific compression rates, allowing parameter budgets to be tailored according to the modality and task. Experiments on vision and language benchmarks reveal that the tensor construction directly impacts performance, sometimes better than standard LoRA under similar parameter counts. |
| title | TensLoRA: Tensor Alternatives for Low-Rank Adaptation |
| topic | Machine Learning Artificial Intelligence 68T I.2.6; I.2.7; I.2.10 |
| url | https://arxiv.org/abs/2509.19391 |