Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.12265 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915604893532160 |
|---|---|
| author | Chen, Junjie Su, Weihang Chu, Zhumin Li, Haitao Zhou, Yujia Yuan, Dingbo Wang, Xudong Zhou, Jun Liu, Yiqun Zhang, Min Ma, Shaoping Ai, Qingyao |
| author_facet | Chen, Junjie Su, Weihang Chu, Zhumin Li, Haitao Zhou, Yujia Yuan, Dingbo Wang, Xudong Zhou, Jun Liu, Yiqun Zhang, Min Ma, Shaoping Ai, Qingyao |
| contents | The rapid development of large language models (LLMs) has highlighted the need for efficient and reliable methods to evaluate their performance. Traditional evaluation methods often face challenges like high costs, limited task formats, dependence on human references, and systematic biases. To address these limitations, we propose Auto-PRE, an automatic LLM evaluation framework inspired by the peer review process. Unlike previous approaches that rely on human annotations, Auto-PRE automatically selects evaluator LLMs based on three core traits: consistency, pertinence, and self-confidence, which correspond to the instruction, content, and response stages, respectively, and collectively cover the entire evaluation process. Experiments on three representative tasks, including summarization, non-factoid QA, and dialogue generation, demonstrate that Auto-PRE achieves state-of-the-art performance while significantly reducing evaluation costs. Furthermore, the structured and scalable design of our automatic qualification exam framework provides valuable insights into automating the evaluation of LLMs-as-judges, paving the way for more advanced LLM-based evaluation frameworks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12265 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Auto-PRE: An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation Chen, Junjie Su, Weihang Chu, Zhumin Li, Haitao Zhou, Yujia Yuan, Dingbo Wang, Xudong Zhou, Jun Liu, Yiqun Zhang, Min Ma, Shaoping Ai, Qingyao Computation and Language The rapid development of large language models (LLMs) has highlighted the need for efficient and reliable methods to evaluate their performance. Traditional evaluation methods often face challenges like high costs, limited task formats, dependence on human references, and systematic biases. To address these limitations, we propose Auto-PRE, an automatic LLM evaluation framework inspired by the peer review process. Unlike previous approaches that rely on human annotations, Auto-PRE automatically selects evaluator LLMs based on three core traits: consistency, pertinence, and self-confidence, which correspond to the instruction, content, and response stages, respectively, and collectively cover the entire evaluation process. Experiments on three representative tasks, including summarization, non-factoid QA, and dialogue generation, demonstrate that Auto-PRE achieves state-of-the-art performance while significantly reducing evaluation costs. Furthermore, the structured and scalable design of our automatic qualification exam framework provides valuable insights into automating the evaluation of LLMs-as-judges, paving the way for more advanced LLM-based evaluation frameworks. |
| title | Auto-PRE: An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.12265 |