Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.04424 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915077966266368 |
|---|---|
| author | Gao, Yicheng Xu, Gonghan Wang, Zhe Cohan, Arman |
| author_facet | Gao, Yicheng Xu, Gonghan Wang, Zhe Cohan, Arman |
| contents | Recent advances in large language models (LLMs) show the potential of using LLMs as evaluators for assessing the quality of text generations from LLMs. However, applying LLM evaluators naively to compare or judge between different systems can lead to unreliable results due to the intrinsic win rate estimation bias of LLM evaluators. In order to mitigate this problem, we propose two calibration methods, Bayesian Win Rate Sampling (BWRS) and Bayesian Dawid-Skene, both of which leverage Bayesian inference to more accurately infer the true win rate of generative language models. We empirically validate our methods on six datasets covering story generation, summarization, and instruction following tasks. We show that both our methods are effective in improving the accuracy of win rate estimation using LLMs as evaluators, offering a promising direction for reliable automatic text quality evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_04424 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Bayesian Calibration of Win Rate Estimation with LLM Evaluators Gao, Yicheng Xu, Gonghan Wang, Zhe Cohan, Arman Computation and Language Artificial Intelligence Recent advances in large language models (LLMs) show the potential of using LLMs as evaluators for assessing the quality of text generations from LLMs. However, applying LLM evaluators naively to compare or judge between different systems can lead to unreliable results due to the intrinsic win rate estimation bias of LLM evaluators. In order to mitigate this problem, we propose two calibration methods, Bayesian Win Rate Sampling (BWRS) and Bayesian Dawid-Skene, both of which leverage Bayesian inference to more accurately infer the true win rate of generative language models. We empirically validate our methods on six datasets covering story generation, summarization, and instruction following tasks. We show that both our methods are effective in improving the accuracy of win rate estimation using LLMs as evaluators, offering a promising direction for reliable automatic text quality evaluation. |
| title | Bayesian Calibration of Win Rate Estimation with LLM Evaluators |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2411.04424 |