Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.04398 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917185323008000 |
|---|---|
| author | Yin, Yuan Venkataramanan, Shashanka Vu, Tuan-Hung Bursuc, Andrei Cord, Matthieu |
| author_facet | Yin, Yuan Venkataramanan, Shashanka Vu, Tuan-Hung Bursuc, Andrei Cord, Matthieu |
| contents | Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, reduce adaptation cost by injecting low-rank updates into pretrained weights. However, LoRA's down-projection is randomly initialized and data-agnostic, discarding potentially useful information. Prior analyses show that this projection changes little during training, while the up-projection carries most of the adaptation, making the random input compression a performance bottleneck. We propose IPA, a feature-aware projection framework that explicitly aims to reconstruct the original input within a reduced hidden space. In the linear case, we instantiate IPA with algorithms approximating top principal components, enabling efficient projector pretraining with negligible inference overhead. Across language and vision benchmarks, IPA consistently improves over LoRA and DoRA, achieving on average 1.5 points higher accuracy on commonsense reasoning and 2.3 points on VTAB-1k, while matching full LoRA performance with roughly half the trainable parameters when the projection is frozen. Code available at https://github.com/valeoai/peft-ipa . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_04398 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation Yin, Yuan Venkataramanan, Shashanka Vu, Tuan-Hung Bursuc, Andrei Cord, Matthieu Machine Learning Artificial Intelligence Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, reduce adaptation cost by injecting low-rank updates into pretrained weights. However, LoRA's down-projection is randomly initialized and data-agnostic, discarding potentially useful information. Prior analyses show that this projection changes little during training, while the up-projection carries most of the adaptation, making the random input compression a performance bottleneck. We propose IPA, a feature-aware projection framework that explicitly aims to reconstruct the original input within a reduced hidden space. In the linear case, we instantiate IPA with algorithms approximating top principal components, enabling efficient projector pretraining with negligible inference overhead. Across language and vision benchmarks, IPA consistently improves over LoRA and DoRA, achieving on average 1.5 points higher accuracy on commonsense reasoning and 2.3 points on VTAB-1k, while matching full LoRA performance with roughly half the trainable parameters when the projection is frozen. Code available at https://github.com/valeoai/peft-ipa . |
| title | IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2509.04398 |