Adaptive Token Biaser: Knowledge Editing via Biasing Key Entities
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909226806280192 |
|---|---|
| author | Bi, Baolong Liu, Shenghua Wang, Yiwei Mei, Lingrui Gao, Hongcheng Xu, Yilong Cheng, Xueqi |
| author_facet | Bi, Baolong Liu, Shenghua Wang, Yiwei Mei, Lingrui Gao, Hongcheng Xu, Yilong Cheng, Xueqi |
| contents | The parametric knowledge memorized by large language models (LLMs) becomes outdated quickly. In-context editing (ICE) is currently the most effective method for updating the knowledge of LLMs. Recent advancements involve enhancing ICE by modifying the decoding strategy, obviating the need for altering internal model structures or adjusting external prompts. However, this enhancement operates across the entire sequence generation, encompassing a plethora of non-critical tokens. In this work, we introduce $\textbf{A}$daptive $\textbf{T}$oken $\textbf{Bias}$er ($\textbf{ATBias}$), a new decoding technique designed to enhance ICE. It focuses on the tokens that are mostly related to knowledge during decoding, biasing their logits by matching key entities related to new and parametric knowledge. Experimental results show that ATBias significantly enhances ICE performance, achieving up to a 32.3% improvement over state-of-the-art ICE methods while incurring only half the latency. ATBias not only improves the knowledge editing capabilities of ICE but can also be widely applied to LLMs with negligible cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12468 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Adaptive Token Biaser: Knowledge Editing via Biasing Key Entities Bi, Baolong Liu, Shenghua Wang, Yiwei Mei, Lingrui Gao, Hongcheng Xu, Yilong Cheng, Xueqi Computation and Language Artificial Intelligence The parametric knowledge memorized by large language models (LLMs) becomes outdated quickly. In-context editing (ICE) is currently the most effective method for updating the knowledge of LLMs. Recent advancements involve enhancing ICE by modifying the decoding strategy, obviating the need for altering internal model structures or adjusting external prompts. However, this enhancement operates across the entire sequence generation, encompassing a plethora of non-critical tokens. In this work, we introduce $\textbf{A}$daptive $\textbf{T}$oken $\textbf{Bias}$er ($\textbf{ATBias}$), a new decoding technique designed to enhance ICE. It focuses on the tokens that are mostly related to knowledge during decoding, biasing their logits by matching key entities related to new and parametric knowledge. Experimental results show that ATBias significantly enhances ICE performance, achieving up to a 32.3% improvement over state-of-the-art ICE methods while incurring only half the latency. ATBias not only improves the knowledge editing capabilities of ICE but can also be widely applied to LLMs with negligible cost. |
| title | Adaptive Token Biaser: Knowledge Editing via Biasing Key Entities |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2406.12468 |