Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.18403 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918042307395584 |
|---|---|
| author | Bavaresco, Anna Bernardi, Raffaella Bertolazzi, Leonardo Elliott, Desmond Fernández, Raquel Gatt, Albert Ghaleb, Esam Giulianelli, Mario Hanna, Michael Koller, Alexander Martins, André F. T. Mondorf, Philipp Neplenbroek, Vera Pezzelle, Sandro Plank, Barbara Schlangen, David Suglia, Alessandro Surikuchi, Aditya K Takmaz, Ece Testoni, Alberto |
| author_facet | Bavaresco, Anna Bernardi, Raffaella Bertolazzi, Leonardo Elliott, Desmond Fernández, Raquel Gatt, Albert Ghaleb, Esam Giulianelli, Mario Hanna, Michael Koller, Alexander Martins, André F. T. Mondorf, Philipp Neplenbroek, Vera Pezzelle, Sandro Plank, Barbara Schlangen, David Suglia, Alessandro Surikuchi, Aditya K Takmaz, Ece Testoni, Alberto |
| contents | There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments, raising questions about the validity of these evaluations, as well as their reproducibility in the case of proprietary models. We provide JUDGE-BENCH, an extensible collection of 20 NLP datasets with human annotations covering a broad range of evaluated properties and types of data, and comprehensively evaluate 11 current LLMs, covering both open-weight and proprietary models, for their ability to replicate the annotations. Our evaluations show substantial variance across models and datasets. Models are reliable evaluators on some tasks, but overall display substantial variability depending on the property being evaluated, the expertise level of the human judges, and whether the language is human or model-generated. We conclude that LLMs should be carefully validated against human judgments before being used as evaluators. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_18403 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks Bavaresco, Anna Bernardi, Raffaella Bertolazzi, Leonardo Elliott, Desmond Fernández, Raquel Gatt, Albert Ghaleb, Esam Giulianelli, Mario Hanna, Michael Koller, Alexander Martins, André F. T. Mondorf, Philipp Neplenbroek, Vera Pezzelle, Sandro Plank, Barbara Schlangen, David Suglia, Alessandro Surikuchi, Aditya K Takmaz, Ece Testoni, Alberto Computation and Language There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments, raising questions about the validity of these evaluations, as well as their reproducibility in the case of proprietary models. We provide JUDGE-BENCH, an extensible collection of 20 NLP datasets with human annotations covering a broad range of evaluated properties and types of data, and comprehensively evaluate 11 current LLMs, covering both open-weight and proprietary models, for their ability to replicate the annotations. Our evaluations show substantial variance across models and datasets. Models are reliable evaluators on some tasks, but overall display substantial variability depending on the property being evaluated, the expertise level of the human judges, and whether the language is human or model-generated. We conclude that LLMs should be carefully validated against human judgments before being used as evaluators. |
| title | LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.18403 |