Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.07967 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929382878085120 |
|---|---|
| author | Ruan, Jie Pu, Xiao Gao, Mingqi Wan, Xiaojun Zhu, Yuesheng |
| author_facet | Ruan, Jie Pu, Xiao Gao, Mingqi Wan, Xiaojun Zhu, Yuesheng |
| contents | Human evaluation is viewed as a reliable evaluation method for NLG which is expensive and time-consuming. To save labor and costs, researchers usually perform human evaluation on a small subset of data sampled from the whole dataset in practice. However, different selection subsets will lead to different rankings of the systems. To give a more correct inter-system ranking and make the gold standard human evaluation more reliable, we propose a Constrained Active Sampling Framework (CASF) for reliable human judgment. CASF operates through a Learner, a Systematic Sampler and a Constrained Controller to select representative samples for getting a more correct inter-system ranking.Experiment results on 137 real NLG evaluation setups with 44 human evaluation metrics across 16 datasets and 5 NLG tasks demonstrate CASF receives 93.18% top-ranked system recognition accuracy and ranks first or ranks second on 90.91% of the human metrics with 0.83 overall inter-system ranking Kendall correlation.Code and data are publicly available online. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07967 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Better than Random: Reliable NLG Human Evaluation with Constrained Active Sampling Ruan, Jie Pu, Xiao Gao, Mingqi Wan, Xiaojun Zhu, Yuesheng Computation and Language Machine Learning Human evaluation is viewed as a reliable evaluation method for NLG which is expensive and time-consuming. To save labor and costs, researchers usually perform human evaluation on a small subset of data sampled from the whole dataset in practice. However, different selection subsets will lead to different rankings of the systems. To give a more correct inter-system ranking and make the gold standard human evaluation more reliable, we propose a Constrained Active Sampling Framework (CASF) for reliable human judgment. CASF operates through a Learner, a Systematic Sampler and a Constrained Controller to select representative samples for getting a more correct inter-system ranking.Experiment results on 137 real NLG evaluation setups with 44 human evaluation metrics across 16 datasets and 5 NLG tasks demonstrate CASF receives 93.18% top-ranked system recognition accuracy and ranks first or ranks second on 90.91% of the human metrics with 0.83 overall inter-system ranking Kendall correlation.Code and data are publicly available online. |
| title | Better than Random: Reliable NLG Human Evaluation with Constrained Active Sampling |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2406.07967 |