Questionable practices in machine learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929568316653568 |
|---|---|
| author | Leech, Gavin Vazquez, Juan J. Kupper, Niclas Yagudin, Misha Aitchison, Laurence |
| author_facet | Leech, Gavin Vazquez, Juan J. Kupper, Niclas Yagudin, Misha Aitchison, Laurence |
| contents | Evaluating modern ML models is hard. The strong incentive for researchers and companies to report a state-of-the-art result on some metric often leads to questionable research practices (QRPs): bad practices which fall short of outright research fraud. We describe 44 such practices which can undermine reported results, giving examples where possible. Our list emphasises the evaluation of large language models (LLMs) on public benchmarks. We also discuss "irreproducible research practices", i.e. decisions that make it difficult or impossible for other researchers to reproduce, build on or audit previous research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_12220 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Questionable practices in machine learning Leech, Gavin Vazquez, Juan J. Kupper, Niclas Yagudin, Misha Aitchison, Laurence Machine Learning Computation and Language Computers and Society Evaluating modern ML models is hard. The strong incentive for researchers and companies to report a state-of-the-art result on some metric often leads to questionable research practices (QRPs): bad practices which fall short of outright research fraud. We describe 44 such practices which can undermine reported results, giving examples where possible. Our list emphasises the evaluation of large language models (LLMs) on public benchmarks. We also discuss "irreproducible research practices", i.e. decisions that make it difficult or impossible for other researchers to reproduce, build on or audit previous research. |
| title | Questionable practices in machine learning |
| topic | Machine Learning Computation and Language Computers and Society |
| url | https://arxiv.org/abs/2407.12220 |