Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.14782 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913175138467840 |
|---|---|
| author | Biderman, Stella Schoelkopf, Hailey Sutawika, Lintang Gao, Leo Tow, Jonathan Abbasi, Baber Aji, Alham Fikri Ammanamanchi, Pawan Sasanka Black, Sidney Clive, Jordan DiPofi, Anthony Etxaniz, Julen Fattori, Benjamin Forde, Jessica Zosa Foster, Charles Hsu, Jeffrey Jaiswal, Mimansa Lee, Wilson Y. Li, Haonan Lovering, Charles Muennighoff, Niklas Pavlick, Ellie Phang, Jason Skowron, Aviya Tan, Samson Tang, Xiangru Wang, Kevin A. Winata, Genta Indra Yvon, François Zou, Andy |
| author_facet | Biderman, Stella Schoelkopf, Hailey Sutawika, Lintang Gao, Leo Tow, Jonathan Abbasi, Baber Aji, Alham Fikri Ammanamanchi, Pawan Sasanka Black, Sidney Clive, Jordan DiPofi, Anthony Etxaniz, Julen Fattori, Benjamin Forde, Jessica Zosa Foster, Charles Hsu, Jeffrey Jaiswal, Mimansa Lee, Wilson Y. Li, Haonan Lovering, Charles Muennighoff, Niklas Pavlick, Ellie Phang, Jason Skowron, Aviya Tan, Samson Tang, Xiangru Wang, Kevin A. Winata, Genta Indra Yvon, François Zou, Andy |
| contents | Reliable evaluation of language models (LMs) remains an open challenge. Re- searchers and engineers face methodological issues such as the sensitivity of models to evaluation setup, difficulty of proper comparisons across methods, and the lack of reproducibility and transparency. Evaluation difficulties are exacer- bated by the fracturing and siloing of information about conventions and common practices. In this paper we draw on three years of experience in evaluating large lan- guage models (LMs) as developers of the popular Language Model Evaluation Harness (lm-eval) (Gao et al., 2023) framework to provide guidance and lessons for the field moving forward. We document a variety of challenges faced by prac- titioners and provide concrete instances where these challenges or the absence of best practices have come into effect. We make recommendations to the field for improving evaluation rigor and confidence, and attempt to codify much of the tacit or folk knowledge surrounding LM evaluation, for a solid ground to move forward. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_14782 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Lessons from the Trenches on Reproducible Evaluation of Language Models Biderman, Stella Schoelkopf, Hailey Sutawika, Lintang Gao, Leo Tow, Jonathan Abbasi, Baber Aji, Alham Fikri Ammanamanchi, Pawan Sasanka Black, Sidney Clive, Jordan DiPofi, Anthony Etxaniz, Julen Fattori, Benjamin Forde, Jessica Zosa Foster, Charles Hsu, Jeffrey Jaiswal, Mimansa Lee, Wilson Y. Li, Haonan Lovering, Charles Muennighoff, Niklas Pavlick, Ellie Phang, Jason Skowron, Aviya Tan, Samson Tang, Xiangru Wang, Kevin A. Winata, Genta Indra Yvon, François Zou, Andy Computation and Language Reliable evaluation of language models (LMs) remains an open challenge. Re- searchers and engineers face methodological issues such as the sensitivity of models to evaluation setup, difficulty of proper comparisons across methods, and the lack of reproducibility and transparency. Evaluation difficulties are exacer- bated by the fracturing and siloing of information about conventions and common practices. In this paper we draw on three years of experience in evaluating large lan- guage models (LMs) as developers of the popular Language Model Evaluation Harness (lm-eval) (Gao et al., 2023) framework to provide guidance and lessons for the field moving forward. We document a variety of challenges faced by prac- titioners and provide concrete instances where these challenges or the absence of best practices have come into effect. We make recommendations to the field for improving evaluation rigor and confidence, and attempt to codify much of the tacit or folk knowledge surrounding LM evaluation, for a solid ground to move forward. |
| title | Lessons from the Trenches on Reproducible Evaluation of Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.14782 |