WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908548700569600 |
|---|---|
| author | Thede, Lukas Roth, Karsten Bethge, Matthias Akata, Zeynep Hartvigsen, Tom |
| author_facet | Thede, Lukas Roth, Karsten Bethge, Matthias Akata, Zeynep Hartvigsen, Tom |
| contents | Keeping large language models factually up-to-date is crucial for deployment, yet costly retraining remains a challenge. Knowledge editing offers a promising alternative, but methods are only tested on small-scale or synthetic edit benchmarks. In this work, we aim to bridge research into lifelong knowledge editing to real-world edits at a practically relevant scale. We first introduce WikiBigEdit; a large-scale benchmark of real-world Wikidata edits, built to automatically extend lifelong for future-proof benchmarking. In its first instance, it includes over 500K question-answer pairs for knowledge editing alongside a comprehensive evaluation pipeline. Finally, we use WikiBigEdit to study existing knowledge editing techniques' ability to incorporate large volumes of real-world facts and contrast their capabilities to generic modification techniques such as retrieval augmentation and continual finetuning to acquire a complete picture of the practical extent of current lifelong knowledge editing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_05683 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs Thede, Lukas Roth, Karsten Bethge, Matthias Akata, Zeynep Hartvigsen, Tom Computation and Language Machine Learning Keeping large language models factually up-to-date is crucial for deployment, yet costly retraining remains a challenge. Knowledge editing offers a promising alternative, but methods are only tested on small-scale or synthetic edit benchmarks. In this work, we aim to bridge research into lifelong knowledge editing to real-world edits at a practically relevant scale. We first introduce WikiBigEdit; a large-scale benchmark of real-world Wikidata edits, built to automatically extend lifelong for future-proof benchmarking. In its first instance, it includes over 500K question-answer pairs for knowledge editing alongside a comprehensive evaluation pipeline. Finally, we use WikiBigEdit to study existing knowledge editing techniques' ability to incorporate large volumes of real-world facts and contrast their capabilities to generic modification techniques such as retrieval augmentation and continual finetuning to acquire a complete picture of the practical extent of current lifelong knowledge editing. |
| title | WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2503.05683 |