Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.13704 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929729744928768 |
|---|---|
| author | Wang, Yicheng Yuan, Jiayi Chuang, Yu-Neng Wang, Zhuoer Liu, Yingchi Cusick, Mark Kulkarni, Param Ji, Zhengping Ibrahim, Yasser Hu, Xia |
| author_facet | Wang, Yicheng Yuan, Jiayi Chuang, Yu-Neng Wang, Zhuoer Liu, Yingchi Cusick, Mark Kulkarni, Param Ji, Zhengping Ibrahim, Yasser Hu, Xia |
| contents | Large Language Models (LLMs) are increasingly serving as evaluators in Natural Language Generation (NLG) tasks; this is often referred to as ``LLM-as-a-judge'' paradigm. However, the capabilities of LLMs in evaluating NLG quality remain underexplored. Current studies depend on human assessments and simple metrics that fail to capture the discernment of LLMs across diverse NLG tasks. To address this gap, we propose the Discernment of Hierarchical Perturbation (DHP) benchmarking framework, which provides quantitative discernment scores for LLMs. This framework leverages hierarchically perturbed text data and statistical tests to systematically measure the NLG evaluation capabilities of LLMs. We re-established six evaluation datasets for this benchmark, covering four NLG tasks: Summarization, Story Completion, Question Answering, and Translation. Our comprehensive benchmarking of five major LLM families provides critical insight into their strengths and limitations as NLG evaluators. Our dataset is available at https://huggingface.co/datasets/YCWANGVINCE/DHP_Benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_13704 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DHP Benchmark: Are LLMs Good NLG Evaluators? Wang, Yicheng Yuan, Jiayi Chuang, Yu-Neng Wang, Zhuoer Liu, Yingchi Cusick, Mark Kulkarni, Param Ji, Zhengping Ibrahim, Yasser Hu, Xia Computation and Language Artificial Intelligence Large Language Models (LLMs) are increasingly serving as evaluators in Natural Language Generation (NLG) tasks; this is often referred to as ``LLM-as-a-judge'' paradigm. However, the capabilities of LLMs in evaluating NLG quality remain underexplored. Current studies depend on human assessments and simple metrics that fail to capture the discernment of LLMs across diverse NLG tasks. To address this gap, we propose the Discernment of Hierarchical Perturbation (DHP) benchmarking framework, which provides quantitative discernment scores for LLMs. This framework leverages hierarchically perturbed text data and statistical tests to systematically measure the NLG evaluation capabilities of LLMs. We re-established six evaluation datasets for this benchmark, covering four NLG tasks: Summarization, Story Completion, Question Answering, and Translation. Our comprehensive benchmarking of five major LLM families provides critical insight into their strengths and limitations as NLG evaluators. Our dataset is available at https://huggingface.co/datasets/YCWANGVINCE/DHP_Benchmark. |
| title | DHP Benchmark: Are LLMs Good NLG Evaluators? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2408.13704 |