Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915666982862848 |
|---|---|
| author | Liu, Tianyu Qi, Jirui He, Paul Bisazza, Arianna Sachan, Mrinmaya Cotterell, Ryan |
| author_facet | Liu, Tianyu Qi, Jirui He, Paul Bisazza, Arianna Sachan, Mrinmaya Cotterell, Ryan |
| contents | Recent work suggests that large language models enhanced with retrieval-augmented generation are easily influenced by the order, in which the retrieved documents are presented to the model when solving tasks such as question answering (QA). However, there is no method to date that exploits this phenomenon to improve generation. We fill this gap. In this study, we show that the pointwise mutual information between a context and a question is an effective gauge for language model performance. Importantly, this gauge does not depend on knowing the answer to the question a priori. Through experiments on two question-answering datasets and a variety of large language models, we find evidence for an empirical correlation between answer accuracy and pointwise mutual information. Additionally, we propose two methods that use the pointwise mutual information between a document and a question as a gauge for selecting and constructing prompts that lead to better performance, whose effectiveness we demonstrate through experimentation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_07773 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation Liu, Tianyu Qi, Jirui He, Paul Bisazza, Arianna Sachan, Mrinmaya Cotterell, Ryan Computation and Language Artificial Intelligence Machine Learning Recent work suggests that large language models enhanced with retrieval-augmented generation are easily influenced by the order, in which the retrieved documents are presented to the model when solving tasks such as question answering (QA). However, there is no method to date that exploits this phenomenon to improve generation. We fill this gap. In this study, we show that the pointwise mutual information between a context and a question is an effective gauge for language model performance. Importantly, this gauge does not depend on knowing the answer to the question a priori. Through experiments on two question-answering datasets and a variety of large language models, we find evidence for an empirical correlation between answer accuracy and pointwise mutual information. Additionally, we propose two methods that use the pointwise mutual information between a document and a question as a gauge for selecting and constructing prompts that lead to better performance, whose effectiveness we demonstrate through experimentation. |
| title | Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2411.07773 |