Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916765687087104 |
|---|---|
| author | Feng, Kehua Ding, Keyan Tan, Hongzhi Ma, Kede Wang, Zhihua Guo, Shuangquan Cheng, Yuzhou Sun, Ge Zheng, Guozhou Zhang, Qiang Chen, Huajun |
| author_facet | Feng, Kehua Ding, Keyan Tan, Hongzhi Ma, Kede Wang, Zhihua Guo, Shuangquan Cheng, Yuzhou Sun, Ge Zheng, Guozhou Zhang, Qiang Chen, Huajun |
| contents | Reliable evaluation of large language models (LLMs) is impeded by two key challenges: objective metrics often fail to reflect human perception of natural language, and exhaustive human labeling is prohibitively expensive. Here, we propose a sample-efficient human evaluation method for LLMs based on the principle of MAximum Discrepancy (MAD) Competition. Our method automatically and adaptively selects a compact set of input instructions that maximize semantic discrepancy between pairs of LLM responses. Human evaluators then perform three-alternative forced choices on these paired responses, which are aggregated into a global ranking using Elo rating. We apply our approach to compare eight widely used LLMs across four tasks: scientific knowledge understanding, mathematical reasoning, creative and functional writing, and code generation and explanation. Experimental results show that our sample-efficient evaluation method recovers "gold-standard" model rankings with a handful of MAD-selected instructions, reveals respective strengths and weaknesses of each LLM, and offers nuanced insights to guide future LLM development. Code is available at https://github.com/weiji-Feng/MAD-Eval . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_08008 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition Feng, Kehua Ding, Keyan Tan, Hongzhi Ma, Kede Wang, Zhihua Guo, Shuangquan Cheng, Yuzhou Sun, Ge Zheng, Guozhou Zhang, Qiang Chen, Huajun Machine Learning Computation and Language Human-Computer Interaction Reliable evaluation of large language models (LLMs) is impeded by two key challenges: objective metrics often fail to reflect human perception of natural language, and exhaustive human labeling is prohibitively expensive. Here, we propose a sample-efficient human evaluation method for LLMs based on the principle of MAximum Discrepancy (MAD) Competition. Our method automatically and adaptively selects a compact set of input instructions that maximize semantic discrepancy between pairs of LLM responses. Human evaluators then perform three-alternative forced choices on these paired responses, which are aggregated into a global ranking using Elo rating. We apply our approach to compare eight widely used LLMs across four tasks: scientific knowledge understanding, mathematical reasoning, creative and functional writing, and code generation and explanation. Experimental results show that our sample-efficient evaluation method recovers "gold-standard" model rankings with a handful of MAD-selected instructions, reveals respective strengths and weaknesses of each LLM, and offers nuanced insights to guide future LLM development. Code is available at https://github.com/weiji-Feng/MAD-Eval . |
| title | Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition |
| topic | Machine Learning Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2404.08008 |