Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.15329 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929620291420160 |
|---|---|
| author | Li, Minzhi Liu, Zhengyuan Deng, Shumin Joty, Shafiq Chen, Nancy F. Kan, Min-Yen |
| author_facet | Li, Minzhi Liu, Zhengyuan Deng, Shumin Joty, Shafiq Chen, Nancy F. Kan, Min-Yen |
| contents | The acceleration of Large Language Models (LLMs) research has opened up new possibilities for evaluating generated texts. They serve as scalable and economical evaluators, but the question of how reliable these evaluators are has emerged as a crucial research question. Prior research efforts in the meta-evaluation of LLMs as judges limit the prompting of an LLM to a single use to obtain a final evaluation decision. They then compute the agreement between LLMs' outputs and human labels. This lacks interpretability in understanding the evaluation capability of LLMs. In light of this challenge, we propose Decompose and Aggregate, which breaks down the evaluation process into different stages based on pedagogical practices. Our experiments illustrate that it not only provides a more interpretable window for how well LLMs evaluate, but also leads to improvements up to 39.6% for different LLMs on a variety of meta-evaluation benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_15329 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation Li, Minzhi Liu, Zhengyuan Deng, Shumin Joty, Shafiq Chen, Nancy F. Kan, Min-Yen Computation and Language The acceleration of Large Language Models (LLMs) research has opened up new possibilities for evaluating generated texts. They serve as scalable and economical evaluators, but the question of how reliable these evaluators are has emerged as a crucial research question. Prior research efforts in the meta-evaluation of LLMs as judges limit the prompting of an LLM to a single use to obtain a final evaluation decision. They then compute the agreement between LLMs' outputs and human labels. This lacks interpretability in understanding the evaluation capability of LLMs. In light of this challenge, we propose Decompose and Aggregate, which breaks down the evaluation process into different stages based on pedagogical practices. Our experiments illustrate that it not only provides a more interpretable window for how well LLMs evaluate, but also leads to improvements up to 39.6% for different LLMs on a variety of meta-evaluation benchmarks. |
| title | DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.15329 |