InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911357213868032 |
|---|---|
| author | Li, Shuaiyi Zhang, Zhisong Deng, Yang Deng, Chenlong Fang, Tianqing Zhang, Hongming Mi, Haitao Yu, Dong Lam, Wai |
| author_facet | Li, Shuaiyi Zhang, Zhisong Deng, Yang Deng, Chenlong Fang, Tianqing Zhang, Hongming Mi, Haitao Yu, Dong Lam, Wai |
| contents | Although existing model editing methods perform well in recalling exact edit facts, they often struggle in complex scenarios that require deeper semantic understanding rather than mere knowledge regurgitation. Leveraging the strong contextual reasoning abilities of large language models (LLMs), in-context learning (ICL) becomes a promising editing method by comprehending edit information through context encoding. However, this method is constrained by the limited context window of LLMs, leading to degraded performance and efficiency as the number of edits increases. To overcome this limitation, we propose InComeS, a flexible framework that enhances LLMs' ability to process editing contexts through explicit compression and selection mechanisms. Specifically, InComeS compresses each editing context into the key-value (KV) cache of a special gist token, enabling efficient handling of multiple edits without being restricted by the model's context window. Furthermore, specialized cross-attention modules are added to dynamically select the most relevant information from the gist pools, enabling adaptive and effective utilization of edit information. We conduct experiments on diverse model editing benchmarks with various editing formats, and the results demonstrate the effectiveness and efficiency of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_22156 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing Li, Shuaiyi Zhang, Zhisong Deng, Yang Deng, Chenlong Fang, Tianqing Zhang, Hongming Mi, Haitao Yu, Dong Lam, Wai Computation and Language Although existing model editing methods perform well in recalling exact edit facts, they often struggle in complex scenarios that require deeper semantic understanding rather than mere knowledge regurgitation. Leveraging the strong contextual reasoning abilities of large language models (LLMs), in-context learning (ICL) becomes a promising editing method by comprehending edit information through context encoding. However, this method is constrained by the limited context window of LLMs, leading to degraded performance and efficiency as the number of edits increases. To overcome this limitation, we propose InComeS, a flexible framework that enhances LLMs' ability to process editing contexts through explicit compression and selection mechanisms. Specifically, InComeS compresses each editing context into the key-value (KV) cache of a special gist token, enabling efficient handling of multiple edits without being restricted by the model's context window. Furthermore, specialized cross-attention modules are added to dynamically select the most relevant information from the gist pools, enabling adaptive and effective utilization of edit information. We conduct experiments on diverse model editing benchmarks with various editing formats, and the results demonstrate the effectiveness and efficiency of our method. |
| title | InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.22156 |