Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.08368 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914461566107648 |
|---|---|
| author | Mohammadabadi, Seyed Mahmoud Sajjadi Ma, Xiaolong Yang, Lei Yan, Feng Zhang, Junshan |
| author_facet | Mohammadabadi, Seyed Mahmoud Sajjadi Ma, Xiaolong Yang, Lei Yan, Feng Zhang, Junshan |
| contents | Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, enable scalable adaptation of foundation models by injecting low-rank adapters. However, their communication and storage costs remain a major bottleneck in resource-constrained settings. We propose SOLAR (Subspace-Oriented Latent Adapter Reparameterization), a post-training compression framework that substantially reduces the communication cost (i.e., the number of parameters to transmit or store) of PEFT adapters. SOLAR expresses each PEFT update as a linear combination of basis vectors formed from the foundation model's singular vectors with controlled random perturbations. By exploiting the subspace similarity (the alignment of principal directions) between the foundation model and task-specific fine-tuned updates, SOLAR decouples the adapter size from PEFT structure and ensures compact yet expressive representations. It is model-agnostic and compatible with existing PEFT methods, including LoRA, AdaLoRA, and other adapter modules. We theoretically establish a bound on the reconstruction error. Experiments on language and vision tasks using LLaMA, GPT, and ViT models demonstrate that SOLAR preserves task performance while significantly reducing model representation sizes, offering an effective and communication-efficient solution for deployment in distributed systems and edge devices. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_08368 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrization Mohammadabadi, Seyed Mahmoud Sajjadi Ma, Xiaolong Yang, Lei Yan, Feng Zhang, Junshan Machine Learning Computation and Language Computer Vision and Pattern Recognition I.2.7; I.2.6 Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, enable scalable adaptation of foundation models by injecting low-rank adapters. However, their communication and storage costs remain a major bottleneck in resource-constrained settings. We propose SOLAR (Subspace-Oriented Latent Adapter Reparameterization), a post-training compression framework that substantially reduces the communication cost (i.e., the number of parameters to transmit or store) of PEFT adapters. SOLAR expresses each PEFT update as a linear combination of basis vectors formed from the foundation model's singular vectors with controlled random perturbations. By exploiting the subspace similarity (the alignment of principal directions) between the foundation model and task-specific fine-tuned updates, SOLAR decouples the adapter size from PEFT structure and ensures compact yet expressive representations. It is model-agnostic and compatible with existing PEFT methods, including LoRA, AdaLoRA, and other adapter modules. We theoretically establish a bound on the reconstruction error. Experiments on language and vision tasks using LLaMA, GPT, and ViT models demonstrate that SOLAR preserves task performance while significantly reducing model representation sizes, offering an effective and communication-efficient solution for deployment in distributed systems and edge devices. |
| title | SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrization |
| topic | Machine Learning Computation and Language Computer Vision and Pattern Recognition I.2.7; I.2.6 |
| url | https://arxiv.org/abs/2604.08368 |