Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.09908 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918115880730624 |
|---|---|
| author | Jin, Pengfei Shu, Peng Song, Sifan Kim, Sekeun Xiao, Qing Chen, Cheng Liu, Tianming Li, Xiang Li, Quanzheng |
| author_facet | Jin, Pengfei Shu, Peng Song, Sifan Kim, Sekeun Xiao, Qing Chen, Cheng Liu, Tianming Li, Xiang Li, Quanzheng |
| contents | Recent advances in parameter-efficient transfer learning have demonstrated the utility of composing LoRA adapters from libraries of pretrained modules. However, most existing approaches rely on simple retrieval heuristics or uniform averaging, which overlook the latent structure of task relationships in representation space. We propose a new framework for adapter reuse that moves beyond retrieval, formulating adapter composition as a geometry-aware sparse reconstruction problem. Specifically, we represent each task by a latent prototype vector derived from the base model's encoder and aim to approximate the target task prototype as a sparse linear combination of retrieved reference prototypes, under an $\ell_1$-regularized optimization objective. The resulting combination weights are then used to blend the corresponding LoRA adapters, yielding a composite adapter tailored to the target task. This formulation not only preserves the local geometric structure of the task representation manifold, but also promotes interpretability and efficient reuse by selecting a minimal set of relevant adapters. We demonstrate the effectiveness of our approach across multiple domains-including medical image segmentation, medical report generation and image synthesis. Our results highlight the benefit of coupling retrieval with latent geometry-aware optimization for improved zero-shot generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_09908 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Beyond Adapter Retrieval: Latent Geometry-Preserving Composition via Sparse Task Projection Jin, Pengfei Shu, Peng Song, Sifan Kim, Sekeun Xiao, Qing Chen, Cheng Liu, Tianming Li, Xiang Li, Quanzheng Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Recent advances in parameter-efficient transfer learning have demonstrated the utility of composing LoRA adapters from libraries of pretrained modules. However, most existing approaches rely on simple retrieval heuristics or uniform averaging, which overlook the latent structure of task relationships in representation space. We propose a new framework for adapter reuse that moves beyond retrieval, formulating adapter composition as a geometry-aware sparse reconstruction problem. Specifically, we represent each task by a latent prototype vector derived from the base model's encoder and aim to approximate the target task prototype as a sparse linear combination of retrieved reference prototypes, under an $\ell_1$-regularized optimization objective. The resulting combination weights are then used to blend the corresponding LoRA adapters, yielding a composite adapter tailored to the target task. This formulation not only preserves the local geometric structure of the task representation manifold, but also promotes interpretability and efficient reuse by selecting a minimal set of relevant adapters. We demonstrate the effectiveness of our approach across multiple domains-including medical image segmentation, medical report generation and image synthesis. Our results highlight the benefit of coupling retrieval with latent geometry-aware optimization for improved zero-shot generalization. |
| title | Beyond Adapter Retrieval: Latent Geometry-Preserving Composition via Sparse Task Projection |
| topic | Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.09908 |