Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929241504874496 |
|---|---|
| author | Hada, Rishav Gumma, Varun de Wynter, Adrian Diddee, Harshita Ahmed, Mohamed Choudhury, Monojit Bali, Kalika Sitaram, Sunayana |
| author_facet | Hada, Rishav Gumma, Varun de Wynter, Adrian Diddee, Harshita Ahmed, Mohamed Choudhury, Monojit Bali, Kalika Sitaram, Sunayana |
| contents | Large Language Models (LLMs) excel in various Natural Language Processing (NLP) tasks, yet their evaluation, particularly in languages beyond the top $20$, remains inadequate due to existing benchmarks and metrics limitations. Employing LLMs as evaluators to rank or score other models' outputs emerges as a viable solution, addressing the constraints tied to human annotators and established benchmarks. In this study, we explore the potential of LLM-based evaluators, specifically GPT-4 in enhancing multilingual evaluation by calibrating them against $20$K human judgments across three text-generation tasks, five metrics, and eight languages. Our analysis reveals a bias in GPT4-based evaluators towards higher scores, underscoring the necessity of calibration with native speaker judgments, especially in low-resource and non-Latin script languages, to ensure accurate evaluation of LLM performance across diverse languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_07462 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation? Hada, Rishav Gumma, Varun de Wynter, Adrian Diddee, Harshita Ahmed, Mohamed Choudhury, Monojit Bali, Kalika Sitaram, Sunayana Computation and Language Large Language Models (LLMs) excel in various Natural Language Processing (NLP) tasks, yet their evaluation, particularly in languages beyond the top $20$, remains inadequate due to existing benchmarks and metrics limitations. Employing LLMs as evaluators to rank or score other models' outputs emerges as a viable solution, addressing the constraints tied to human annotators and established benchmarks. In this study, we explore the potential of LLM-based evaluators, specifically GPT-4 in enhancing multilingual evaluation by calibrating them against $20$K human judgments across three text-generation tasks, five metrics, and eight languages. Our analysis reveals a bias in GPT4-based evaluators towards higher scores, underscoring the necessity of calibration with native speaker judgments, especially in low-resource and non-Latin script languages, to ensure accurate evaluation of LLM performance across diverse languages. |
| title | Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2309.07462 |