Data-driven Clustering and Merging of Adapters for On-device Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914277631197184 |
|---|---|
| author | Bohdal, Ondrej Ceritli, Taha Ozay, Mete Moon, Jijoong Lee, Kyeng-Hun Ko, Hyeonmok Michieli, Umberto |
| author_facet | Bohdal, Ondrej Ceritli, Taha Ozay, Mete Moon, Jijoong Lee, Kyeng-Hun Ko, Hyeonmok Michieli, Umberto |
| contents | On-device large language models commonly employ task-specific adapters (e.g., LoRAs) to deliver strong performance on downstream tasks. While storing all available adapters is impractical due to memory constraints, mobile devices typically have sufficient capacity to store a limited number of these parameters. This raises a critical challenge: how to select representative adapters that generalize well across multiple tasks - a problem that remains unexplored in existing literature. We propose a novel method D2C for adapter clustering that leverages minimal task-specific examples (e.g., 10 per task) and employs an iterative optimization process to refine cluster assignments. The adapters within each cluster are merged, creating multi-task adapters deployable on resource-constrained devices. Experimental results demonstrate that our method effectively boosts performance for considered storage budgets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_17441 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Data-driven Clustering and Merging of Adapters for On-device Large Language Models Bohdal, Ondrej Ceritli, Taha Ozay, Mete Moon, Jijoong Lee, Kyeng-Hun Ko, Hyeonmok Michieli, Umberto Machine Learning Artificial Intelligence Computation and Language On-device large language models commonly employ task-specific adapters (e.g., LoRAs) to deliver strong performance on downstream tasks. While storing all available adapters is impractical due to memory constraints, mobile devices typically have sufficient capacity to store a limited number of these parameters. This raises a critical challenge: how to select representative adapters that generalize well across multiple tasks - a problem that remains unexplored in existing literature. We propose a novel method D2C for adapter clustering that leverages minimal task-specific examples (e.g., 10 per task) and employs an iterative optimization process to refine cluster assignments. The adapters within each cluster are merged, creating multi-task adapters deployable on resource-constrained devices. Experimental results demonstrate that our method effectively boosts performance for considered storage budgets. |
| title | Data-driven Clustering and Merging of Adapters for On-device Large Language Models |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2601.17441 |