Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapse
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909028838277120 |
|---|---|
| author | Zhang, Chi Zhang, Mengqi Ye, Xiaotian Cheng, Runxi Zhou, Zisheng Zhou, Ying Ren, Pengjie Chen, Zhumin |
| author_facet | Zhang, Chi Zhang, Mengqi Ye, Xiaotian Cheng, Runxi Zhou, Zisheng Zhou, Ying Ren, Pengjie Chen, Zhumin |
| contents | Sequential knowledge editing in large language models often causes catastrophic collapse of the model's general abilities, especially for parameter-modifying methods. Existing approaches mitigate this issue through heuristic constraints on parameter updates, yet the mechanisms underlying such degradation remain insufficiently understood. In this work, we present a spectral analysis of sequential knowledge editing and show that a model's general abilities are closely associated with dominant singular directions of pretrained weight matrices. These directions are highly sensitive to perturbations and are progressively disrupted by repeated edits, closely tracking the collapse in both editing efficacy and general performance. Building on this insight, we propose REVIVE, a plug-and-play framework that stabilizes sequential editing by explicitly preserving the dominant singular subspace. REVIVE represents parameter updates in the spectral basis of the original weights and filters components that would interfere with the protected region. Extensive experiments across multiple models and benchmarks show that REVIVE consistently improves editing efficacy while substantially preserving general abilities under long-horizon sequential editing, including extreme settings with up to 20,000 edits. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_11042 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapse Zhang, Chi Zhang, Mengqi Ye, Xiaotian Cheng, Runxi Zhou, Zisheng Zhou, Ying Ren, Pengjie Chen, Zhumin Computation and Language Artificial Intelligence Sequential knowledge editing in large language models often causes catastrophic collapse of the model's general abilities, especially for parameter-modifying methods. Existing approaches mitigate this issue through heuristic constraints on parameter updates, yet the mechanisms underlying such degradation remain insufficiently understood. In this work, we present a spectral analysis of sequential knowledge editing and show that a model's general abilities are closely associated with dominant singular directions of pretrained weight matrices. These directions are highly sensitive to perturbations and are progressively disrupted by repeated edits, closely tracking the collapse in both editing efficacy and general performance. Building on this insight, we propose REVIVE, a plug-and-play framework that stabilizes sequential editing by explicitly preserving the dominant singular subspace. REVIVE represents parameter updates in the spectral basis of the original weights and filters components that would interfere with the protected region. Extensive experiments across multiple models and benchmarks show that REVIVE consistently improves editing efficacy while substantially preserving general abilities under long-horizon sequential editing, including extreme settings with up to 20,000 edits. |
| title | Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapse |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2601.11042 |