ScEdit: Script-based Assessment of Knowledge Editing
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909631565004800 |
|---|---|
| author | Li, Xinye Zheng, Zunwen Zhang, Qian Zhuang, Dekai Kang, Jiabao Xu, Liyan Liu, Qingbin Chen, Xi Tu, Zhiying Chu, Dianhui Sui, Dianbo |
| author_facet | Li, Xinye Zheng, Zunwen Zhang, Qian Zhuang, Dekai Kang, Jiabao Xu, Liyan Liu, Qingbin Chen, Xi Tu, Zhiying Chu, Dianhui Sui, Dianbo |
| contents | Knowledge Editing (KE) has gained increasing attention, yet current KE tasks remain relatively simple. Under current evaluation frameworks, many editing methods achieve exceptionally high scores, sometimes nearing perfection. However, few studies integrate KE into real-world application scenarios (e.g., recent interest in LLM-as-agent). To support our analysis, we introduce a novel script-based benchmark -- ScEdit (Script-based Knowledge Editing Benchmark) -- which encompasses both counterfactual and temporal edits. We integrate token-level and text-level evaluation methods, comprehensively analyzing existing KE techniques. The benchmark extends traditional fact-based ("What"-type question) evaluation to action-based ("How"-type question) evaluation. We observe that all KE methods exhibit a drop in performance on established metrics and face challenges on text-level metrics, indicating a challenging task. Our benchmark is available at https://github.com/asdfo123/ScEdit. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23291 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ScEdit: Script-based Assessment of Knowledge Editing Li, Xinye Zheng, Zunwen Zhang, Qian Zhuang, Dekai Kang, Jiabao Xu, Liyan Liu, Qingbin Chen, Xi Tu, Zhiying Chu, Dianhui Sui, Dianbo Computation and Language Knowledge Editing (KE) has gained increasing attention, yet current KE tasks remain relatively simple. Under current evaluation frameworks, many editing methods achieve exceptionally high scores, sometimes nearing perfection. However, few studies integrate KE into real-world application scenarios (e.g., recent interest in LLM-as-agent). To support our analysis, we introduce a novel script-based benchmark -- ScEdit (Script-based Knowledge Editing Benchmark) -- which encompasses both counterfactual and temporal edits. We integrate token-level and text-level evaluation methods, comprehensively analyzing existing KE techniques. The benchmark extends traditional fact-based ("What"-type question) evaluation to action-based ("How"-type question) evaluation. We observe that all KE methods exhibit a drop in performance on established metrics and face challenges on text-level metrics, indicating a challenging task. Our benchmark is available at https://github.com/asdfo123/ScEdit. |
| title | ScEdit: Script-based Assessment of Knowledge Editing |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.23291 |