An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914059571429376 |
|---|---|
| author | Zhu, Kun Feng, Xiaocheng Du, Xiyuan Gu, Yuxuan Yu, Weijiang Wang, Haotian Chen, Qianglong Chu, Zheng Chen, Jingchang Qin, Bing |
| author_facet | Zhu, Kun Feng, Xiaocheng Du, Xiyuan Gu, Yuxuan Yu, Weijiang Wang, Haotian Chen, Qianglong Chu, Zheng Chen, Jingchang Qin, Bing |
| contents | Retrieval-augmented generation integrates the capabilities of large language models with relevant information retrieved from an extensive corpus, yet encounters challenges when confronted with real-world noisy data. One recent solution is to train a filter module to find relevant content but only achieve suboptimal noise compression. In this paper, we propose to introduce the information bottleneck theory into retrieval-augmented generation. Our approach involves the filtration of noise by simultaneously maximizing the mutual information between compression and ground output, while minimizing the mutual information between compression and retrieved passage. In addition, we derive the formula of information bottleneck to facilitate its application in novel comprehensive evaluations, the selection of supervised fine-tuning data, and the construction of reinforcement learning rewards. Experimental results demonstrate that our approach achieves significant improvements across various question answering datasets, not only in terms of the correctness of answer generation but also in the conciseness with $2.5\%$ compression rate. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_01549 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation Zhu, Kun Feng, Xiaocheng Du, Xiyuan Gu, Yuxuan Yu, Weijiang Wang, Haotian Chen, Qianglong Chu, Zheng Chen, Jingchang Qin, Bing Computation and Language Artificial Intelligence Retrieval-augmented generation integrates the capabilities of large language models with relevant information retrieved from an extensive corpus, yet encounters challenges when confronted with real-world noisy data. One recent solution is to train a filter module to find relevant content but only achieve suboptimal noise compression. In this paper, we propose to introduce the information bottleneck theory into retrieval-augmented generation. Our approach involves the filtration of noise by simultaneously maximizing the mutual information between compression and ground output, while minimizing the mutual information between compression and retrieved passage. In addition, we derive the formula of information bottleneck to facilitate its application in novel comprehensive evaluations, the selection of supervised fine-tuning data, and the construction of reinforcement learning rewards. Experimental results demonstrate that our approach achieves significant improvements across various question answering datasets, not only in terms of the correctness of answer generation but also in the conciseness with $2.5\%$ compression rate. |
| title | An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2406.01549 |