Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915540070563840 |
|---|---|
| author | Zeng, Siqi He, Yifei Liu, Meitong You, Weiqiu Hao, Yifan Tsai, Yao-Hung Hubert Yamada, Makoto Zhao, Han |
| author_facet | Zeng, Siqi He, Yifei Liu, Meitong You, Weiqiu Hao, Yifan Tsai, Yao-Hung Hubert Yamada, Makoto Zhao, Han |
| contents | Task arithmetic, representing downstream tasks through linear operations on task vectors, has emerged as a simple yet powerful paradigm for transferring knowledge across diverse settings. However, maintaining a large collection of task vectors introduces scalability challenges in both storage and computation. We propose Task Vector Bases, a framework compressing $T$ task vectors into $M < T$ basis vectors while preserving the functionality of task arithmetic. By representing each task vector as a structured linear combination of basis atoms, our approach supports standard operations such as addition, negation, as well as more advanced arithmetic ones. The framework is orthogonal to other efficiency-oriented improvements in task arithmetic and can be used in combination with them. We provide theoretical analysis showing that basis compression retains addition generalization guarantees and enables principled unlearning, with error bounds depending on reconstruction quality. Empirically, our proposed basis construction methods consistently outperform heuristic basis construction baselines and, in some cases, even surpass the performance of full task vector collections across diverse downstream applications while reducing storage and computational requirements. The code is available at https://github.com/uiuctml/TaskVectorBasis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_01015 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic Zeng, Siqi He, Yifei Liu, Meitong You, Weiqiu Hao, Yifan Tsai, Yao-Hung Hubert Yamada, Makoto Zhao, Han Machine Learning Task arithmetic, representing downstream tasks through linear operations on task vectors, has emerged as a simple yet powerful paradigm for transferring knowledge across diverse settings. However, maintaining a large collection of task vectors introduces scalability challenges in both storage and computation. We propose Task Vector Bases, a framework compressing $T$ task vectors into $M < T$ basis vectors while preserving the functionality of task arithmetic. By representing each task vector as a structured linear combination of basis atoms, our approach supports standard operations such as addition, negation, as well as more advanced arithmetic ones. The framework is orthogonal to other efficiency-oriented improvements in task arithmetic and can be used in combination with them. We provide theoretical analysis showing that basis compression retains addition generalization guarantees and enables principled unlearning, with error bounds depending on reconstruction quality. Empirically, our proposed basis construction methods consistently outperform heuristic basis construction baselines and, in some cases, even surpass the performance of full task vector collections across diverse downstream applications while reducing storage and computational requirements. The code is available at https://github.com/uiuctml/TaskVectorBasis. |
| title | Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2502.01015 |