Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.01474 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916188839215104 |
|---|---|
| author | Riley, Parker Deutsch, Daniel Foster, George Ratnakar, Viresh Dabirmoghaddam, Ali Freitag, Markus |
| author_facet | Riley, Parker Deutsch, Daniel Foster, George Ratnakar, Viresh Dabirmoghaddam, Ali Freitag, Markus |
| contents | Reliable human evaluation is critical to the development of successful natural language generation models, but achieving it is notoriously difficult. Stability is a crucial requirement when ranking systems by quality: consistent ranking of systems across repeated evaluations is not just desirable, but essential. Without it, there is no reliable foundation for hill-climbing or product launch decisions. In this paper, we use machine translation and its state-of-the-art human evaluation framework, MQM, as a case study to understand how to set up reliable human evaluations that yield stable conclusions. We investigate the optimal configurations for item allocation to raters, number of ratings per item, and score normalization. Our study on two language pairs provides concrete recommendations for designing replicable human evaluation studies. We also collect and release the largest publicly available dataset of multi-segment translations rated by multiple professional translators, consisting of nearly 140,000 segment annotations across two language pairs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_01474 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Finding Replicable Human Evaluations via Stable Ranking Probability Riley, Parker Deutsch, Daniel Foster, George Ratnakar, Viresh Dabirmoghaddam, Ali Freitag, Markus Computation and Language Reliable human evaluation is critical to the development of successful natural language generation models, but achieving it is notoriously difficult. Stability is a crucial requirement when ranking systems by quality: consistent ranking of systems across repeated evaluations is not just desirable, but essential. Without it, there is no reliable foundation for hill-climbing or product launch decisions. In this paper, we use machine translation and its state-of-the-art human evaluation framework, MQM, as a case study to understand how to set up reliable human evaluations that yield stable conclusions. We investigate the optimal configurations for item allocation to raters, number of ratings per item, and score normalization. Our study on two language pairs provides concrete recommendations for designing replicable human evaluation studies. We also collect and release the largest publicly available dataset of multi-segment translations rated by multiple professional translators, consisting of nearly 140,000 segment annotations across two language pairs. |
| title | Finding Replicable Human Evaluations via Stable Ranking Probability |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.01474 |