Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.02421 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915603254607872 |
|---|---|
| author | Nobari, Amin Heyrani Alim, Kaveh ArjomandBigdeli, Ali Srivastava, Akash Ahmed, Faez Azizan, Navid |
| author_facet | Nobari, Amin Heyrani Alim, Kaveh ArjomandBigdeli, Ali Srivastava, Akash Ahmed, Faez Azizan, Navid |
| contents | Model merging, a method that combines the parameters and embeddings of multiple fine-tuned large language models (LLMs), offers a promising approach to enhance model performance across various tasks while maintaining computational efficiency. This paper introduces Activation-Informed Merging (AIM), a technique that integrates the information from the activation space of LLMs into the merging process to improve performance and robustness. AIM is designed as a flexible, complementary solution that is applicable to any existing merging method. It aims to preserve critical weights from the base model, drawing on principles from continual learning (CL) and model compression. Utilizing a task-agnostic calibration set, AIM selectively prioritizes essential weights during merging. We empirically demonstrate that AIM significantly enhances the performance of merged models across multiple benchmarks. Our findings suggest that considering the activation-space information can provide substantial advancements in the model merging strategies for LLMs, with up to a 40% increase in benchmark performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_02421 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Activation-Informed Merging of Large Language Models Nobari, Amin Heyrani Alim, Kaveh ArjomandBigdeli, Ali Srivastava, Akash Ahmed, Faez Azizan, Navid Computation and Language Artificial Intelligence Model merging, a method that combines the parameters and embeddings of multiple fine-tuned large language models (LLMs), offers a promising approach to enhance model performance across various tasks while maintaining computational efficiency. This paper introduces Activation-Informed Merging (AIM), a technique that integrates the information from the activation space of LLMs into the merging process to improve performance and robustness. AIM is designed as a flexible, complementary solution that is applicable to any existing merging method. It aims to preserve critical weights from the base model, drawing on principles from continual learning (CL) and model compression. Utilizing a task-agnostic calibration set, AIM selectively prioritizes essential weights during merging. We empirically demonstrate that AIM significantly enhances the performance of merged models across multiple benchmarks. Our findings suggest that considering the activation-space information can provide substantial advancements in the model merging strategies for LLMs, with up to a 40% increase in benchmark performance. |
| title | Activation-Informed Merging of Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.02421 |