Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915001842794496 |
|---|---|
| author | Kang, Minki Hwang, Sung Ju Lee, Gibbeum Cho, Jaewoong |
| author_facet | Kang, Minki Hwang, Sung Ju Lee, Gibbeum Cho, Jaewoong |
| contents | As Large Language Models (LLMs) are increasingly deployed in specialized domains with continuously evolving knowledge, the need for timely and precise knowledge injection has become essential. Fine-tuning with paraphrased data is a common approach to enhance knowledge injection, yet it faces two significant challenges: high computational costs due to repetitive external model usage and limited sample diversity. To this end, we introduce LaPael, a latent-level paraphrasing method that applies input-dependent noise to early LLM layers. This approach enables diverse and semantically consistent augmentations directly within the model. Furthermore, it eliminates the recurring costs of paraphrase generation for each knowledge update. Our extensive experiments on question-answering benchmarks demonstrate that LaPael improves knowledge injection over standard fine-tuning and existing noise-based approaches. Additionally, combining LaPael with data-level paraphrasing further enhances performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_00686 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models Kang, Minki Hwang, Sung Ju Lee, Gibbeum Cho, Jaewoong Computation and Language Artificial Intelligence As Large Language Models (LLMs) are increasingly deployed in specialized domains with continuously evolving knowledge, the need for timely and precise knowledge injection has become essential. Fine-tuning with paraphrased data is a common approach to enhance knowledge injection, yet it faces two significant challenges: high computational costs due to repetitive external model usage and limited sample diversity. To this end, we introduce LaPael, a latent-level paraphrasing method that applies input-dependent noise to early LLM layers. This approach enables diverse and semantically consistent augmentations directly within the model. Furthermore, it eliminates the recurring costs of paraphrase generation for each knowledge update. Our extensive experiments on question-answering benchmarks demonstrate that LaPael improves knowledge injection over standard fine-tuning and existing noise-based approaches. Additionally, combining LaPael with data-level paraphrasing further enhances performance. |
| title | Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2411.00686 |