Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915939095674880 |
|---|---|
| author | Liu, Xuyuan Chen, Shengyu Dong, Xinshuai Liu, Yanchi Zhao, Xujiang Wang, Haoyu Yan, Yujun Chen, Haifeng Chen, Zhengzhang |
| author_facet | Liu, Xuyuan Chen, Shengyu Dong, Xinshuai Liu, Yanchi Zhao, Xujiang Wang, Haoyu Yan, Yujun Chen, Haifeng Chen, Zhengzhang |
| contents | Large language models (LLMs) often produce incorrect or outdated content after being employed. Efficient and accurate knowledge updates without costly retraining are a major challenge. This problem is particularly challenging in lifelong settings, where complex, unstructured knowledge must coexist without interference. We introduce RILKE (Representation Intervention for Lifelong KnowledgE Control), a robust and scalable method that treats knowledge control as interventions within the model's representation space. Leveraging representation-space expressiveness, we identify two key properties enabling RILKE to achieve fine-grained control over complex, unstructured knowledge while maintaining general utility with frozen base weights. During training, RILKE learns paraphrase-robust and edit-localized modules that limit each update to a low-dimensional subspace to minimize cross-edit interference. At inference, a query-adaptive router selects the appropriate module to guide the model's generation. Across LLaMA and Qwen models, RILKE scales effectively to large-scale benchmarks, demonstrating high edit success and strong paraphrase generalization while preserving general utility with modest memory overhead. These results show RILKE is an effective and scalable solution for lifelong knowledge control in LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_20892 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs Liu, Xuyuan Chen, Shengyu Dong, Xinshuai Liu, Yanchi Zhao, Xujiang Wang, Haoyu Yan, Yujun Chen, Haifeng Chen, Zhengzhang Artificial Intelligence Large language models (LLMs) often produce incorrect or outdated content after being employed. Efficient and accurate knowledge updates without costly retraining are a major challenge. This problem is particularly challenging in lifelong settings, where complex, unstructured knowledge must coexist without interference. We introduce RILKE (Representation Intervention for Lifelong KnowledgE Control), a robust and scalable method that treats knowledge control as interventions within the model's representation space. Leveraging representation-space expressiveness, we identify two key properties enabling RILKE to achieve fine-grained control over complex, unstructured knowledge while maintaining general utility with frozen base weights. During training, RILKE learns paraphrase-robust and edit-localized modules that limit each update to a low-dimensional subspace to minimize cross-edit interference. At inference, a query-adaptive router selects the appropriate module to guide the model's generation. Across LLaMA and Qwen models, RILKE scales effectively to large-scale benchmarks, demonstrating high edit success and strong paraphrase generalization while preserving general utility with modest memory overhead. These results show RILKE is an effective and scalable solution for lifelong knowledge control in LLMs. |
| title | Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2511.20892 |