Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.20479 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909865231777792 |
|---|---|
| author | Wang, Bowen Wan, Haiyuan Shi, Liwen Yang, Chen He, Peng Ma, Yue Han, Haochen Li, Wenhao Tan, Tiao Li, Yongjian Liu, Fangming Gong, Yifan Zhang, Sheng |
| author_facet | Wang, Bowen Wan, Haiyuan Shi, Liwen Yang, Chen He, Peng Ma, Yue Han, Haochen Li, Wenhao Tan, Tiao Li, Yongjian Liu, Fangming Gong, Yifan Zhang, Sheng |
| contents | We unveil that internal representations in large language models (LLMs) serve as reliable proxies of learned knowledge, and propose RECALL, a novel representation-aware model merging framework for continual learning without access to historical data. RECALL computes inter-model similarity from layer-wise hidden representations over clustered typical samples, and performs adaptive, hierarchical parameter fusion to align knowledge across models. This design enables the preservation of domain-general features in shallow layers while allowing task-specific adaptation in deeper layers. Unlike prior methods that require task labels or incur performance trade-offs, RECALL achieves seamless multi-domain integration and strong resistance to catastrophic forgetting. Extensive experiments across five NLP tasks and multiple continual learning scenarios show that RECALL outperforms baselines in both knowledge retention and generalization, providing a scalable and data-free solution for evolving LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_20479 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging Wang, Bowen Wan, Haiyuan Shi, Liwen Yang, Chen He, Peng Ma, Yue Han, Haochen Li, Wenhao Tan, Tiao Li, Yongjian Liu, Fangming Gong, Yifan Zhang, Sheng Computation and Language Artificial Intelligence We unveil that internal representations in large language models (LLMs) serve as reliable proxies of learned knowledge, and propose RECALL, a novel representation-aware model merging framework for continual learning without access to historical data. RECALL computes inter-model similarity from layer-wise hidden representations over clustered typical samples, and performs adaptive, hierarchical parameter fusion to align knowledge across models. This design enables the preservation of domain-general features in shallow layers while allowing task-specific adaptation in deeper layers. Unlike prior methods that require task labels or incur performance trade-offs, RECALL achieves seamless multi-domain integration and strong resistance to catastrophic forgetting. Extensive experiments across five NLP tasks and multiple continual learning scenarios show that RECALL outperforms baselines in both knowledge retention and generalization, providing a scalable and data-free solution for evolving LLMs. |
| title | RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.20479 |