Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913706737139712 |
|---|---|
| author | Schaeffer, Rylan Koura, Punit Singh Tang, Binh Subramanian, Ranjan Singh, Aaditya K Mihaylov, Todor Bhargava, Prajjwal Madaan, Lovish Chatterji, Niladri S. Goswami, Vedanuj Edunov, Sergey Hupkes, Dieuwke Koyejo, Sanmi Narang, Sharan |
| author_facet | Schaeffer, Rylan Koura, Punit Singh Tang, Binh Subramanian, Ranjan Singh, Aaditya K Mihaylov, Todor Bhargava, Prajjwal Madaan, Lovish Chatterji, Niladri S. Goswami, Vedanuj Edunov, Sergey Hupkes, Dieuwke Koyejo, Sanmi Narang, Sharan |
| contents | The explosion of high-performing conversational language models (LMs) has spurred a shift from classic natural language processing (NLP) benchmarks to expensive, time-consuming and noisy human evaluations - yet the relationship between these two evaluation strategies remains hazy. In this paper, we conduct a large-scale study of four Chat Llama 2 models, comparing their performance on 160 standard NLP benchmarks (e.g., MMLU, ARC, BIG-Bench Hard) against extensive human preferences on more than 11k single-turn and 2k multi-turn dialogues from over 2k human annotators. Our findings are striking: most NLP benchmarks strongly correlate with human evaluations, suggesting that cheaper, automated metrics can serve as surprisingly reliable predictors of human preferences. Three human evaluations, such as adversarial dishonesty and safety, are anticorrelated with NLP benchmarks, while two are uncorrelated. Moreover, through overparameterized linear regressions, we show that NLP scores can accurately predict human evaluations across different model scales, offering a path to reduce costly human annotation without sacrificing rigor. Overall, our results affirm the continued value of classic benchmarks and illuminate how to harness them to anticipate real-world user satisfaction - pointing to how NLP benchmarks can be leveraged to meet evaluation needs of our new era of conversational AI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_18339 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarks Schaeffer, Rylan Koura, Punit Singh Tang, Binh Subramanian, Ranjan Singh, Aaditya K Mihaylov, Todor Bhargava, Prajjwal Madaan, Lovish Chatterji, Niladri S. Goswami, Vedanuj Edunov, Sergey Hupkes, Dieuwke Koyejo, Sanmi Narang, Sharan Computation and Language Machine Learning The explosion of high-performing conversational language models (LMs) has spurred a shift from classic natural language processing (NLP) benchmarks to expensive, time-consuming and noisy human evaluations - yet the relationship between these two evaluation strategies remains hazy. In this paper, we conduct a large-scale study of four Chat Llama 2 models, comparing their performance on 160 standard NLP benchmarks (e.g., MMLU, ARC, BIG-Bench Hard) against extensive human preferences on more than 11k single-turn and 2k multi-turn dialogues from over 2k human annotators. Our findings are striking: most NLP benchmarks strongly correlate with human evaluations, suggesting that cheaper, automated metrics can serve as surprisingly reliable predictors of human preferences. Three human evaluations, such as adversarial dishonesty and safety, are anticorrelated with NLP benchmarks, while two are uncorrelated. Moreover, through overparameterized linear regressions, we show that NLP scores can accurately predict human evaluations across different model scales, offering a path to reduce costly human annotation without sacrificing rigor. Overall, our results affirm the continued value of classic benchmarks and illuminate how to harness them to anticipate real-world user satisfaction - pointing to how NLP benchmarks can be leveraged to meet evaluation needs of our new era of conversational AI. |
| title | Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarks |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2502.18339 |