CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912719432581120 |
|---|---|
| author | Yao, Yunzhi Fang, Jizhan Gu, Jia-Chen Zhang, Ningyu Deng, Shumin Chen, Huajun Peng, Nanyun |
| author_facet | Yao, Yunzhi Fang, Jizhan Gu, Jia-Chen Zhang, Ningyu Deng, Shumin Chen, Huajun Peng, Nanyun |
| contents | Knowledge Editing (KE) enables the modification of outdated or incorrect information in large language models (LLMs). While existing KE methods can update isolated facts, they often fail to generalize these updates to multi-hop reasoning tasks that rely on the modified knowledge. Through an analysis of reasoning circuits -- the neural pathways LLMs use for knowledge-based inference, we find that current layer-localized KE approaches (e.g., MEMIT, WISE), which edit only single or a few model layers, inadequately integrate updated knowledge into these reasoning pathways. To address this limitation, we present CaKE (Circuit-aware Knowledge Editing), a novel method that enhances the effective integration of updated knowledge in LLMs. By only leveraging a few curated data samples guided by our circuit-based analysis, CaKE stimulates the model to develop appropriate reasoning circuits for newly incorporated knowledge. Experiments show that CaKE enables more accurate and consistent use of edited knowledge across related reasoning tasks, achieving an average improvement of 20% in multi-hop reasoning accuracy on the MQuAKE dataset while requiring less memory than existing KE methods. We release the code and data in https://github.com/zjunlp/CaKE. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_16356 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners Yao, Yunzhi Fang, Jizhan Gu, Jia-Chen Zhang, Ningyu Deng, Shumin Chen, Huajun Peng, Nanyun Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Information Retrieval Machine Learning Knowledge Editing (KE) enables the modification of outdated or incorrect information in large language models (LLMs). While existing KE methods can update isolated facts, they often fail to generalize these updates to multi-hop reasoning tasks that rely on the modified knowledge. Through an analysis of reasoning circuits -- the neural pathways LLMs use for knowledge-based inference, we find that current layer-localized KE approaches (e.g., MEMIT, WISE), which edit only single or a few model layers, inadequately integrate updated knowledge into these reasoning pathways. To address this limitation, we present CaKE (Circuit-aware Knowledge Editing), a novel method that enhances the effective integration of updated knowledge in LLMs. By only leveraging a few curated data samples guided by our circuit-based analysis, CaKE stimulates the model to develop appropriate reasoning circuits for newly incorporated knowledge. Experiments show that CaKE enables more accurate and consistent use of edited knowledge across related reasoning tasks, achieving an average improvement of 20% in multi-hop reasoning accuracy on the MQuAKE dataset while requiring less memory than existing KE methods. We release the code and data in https://github.com/zjunlp/CaKE. |
| title | CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2503.16356 |