CORD: Balancing COnsistency and Rank Distillation for Robust Retrieval-Augmented Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909434000703488 |
|---|---|
| author | Lee, Youngwon Hwang, Seung-won Campos, Daniel Graliński, Filip Yao, Zhewei He, Yuxiong |
| author_facet | Lee, Youngwon Hwang, Seung-won Campos, Daniel Graliński, Filip Yao, Zhewei He, Yuxiong |
| contents | With the adoption of retrieval-augmented generation (RAG), large language models (LLMs) are expected to ground their generation to the retrieved contexts. Yet, this is hindered by position bias of LLMs, failing to evenly attend to all contexts. Previous work has addressed this by synthesizing contexts with perturbed positions of gold segment, creating a position-diversified train set. We extend this intuition to propose consistency regularization with augmentation and distillation. First, we augment each training instance with its position perturbation to encourage consistent predictions, regardless of ordering. We also distill behaviors of this pair, although it can be counterproductive in certain RAG scenarios where the given order from the retriever is crucial for generation quality. We thus propose CORD, balancing COnsistency and Rank Distillation. CORD adaptively samples noise-controlled perturbations from an interpolation space, ensuring both consistency and respect for the rank prior. Empirical results show this balance enables CORD to outperform consistently in diverse RAG benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_14581 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CORD: Balancing COnsistency and Rank Distillation for Robust Retrieval-Augmented Generation Lee, Youngwon Hwang, Seung-won Campos, Daniel Graliński, Filip Yao, Zhewei He, Yuxiong Computation and Language With the adoption of retrieval-augmented generation (RAG), large language models (LLMs) are expected to ground their generation to the retrieved contexts. Yet, this is hindered by position bias of LLMs, failing to evenly attend to all contexts. Previous work has addressed this by synthesizing contexts with perturbed positions of gold segment, creating a position-diversified train set. We extend this intuition to propose consistency regularization with augmentation and distillation. First, we augment each training instance with its position perturbation to encourage consistent predictions, regardless of ordering. We also distill behaviors of this pair, although it can be counterproductive in certain RAG scenarios where the given order from the retriever is crucial for generation quality. We thus propose CORD, balancing COnsistency and Rank Distillation. CORD adaptively samples noise-controlled perturbations from an interpolation space, ensuring both consistency and respect for the rank prior. Empirical results show this balance enables CORD to outperform consistently in diverse RAG benchmarks. |
| title | CORD: Balancing COnsistency and Rank Distillation for Robust Retrieval-Augmented Generation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.14581 |