MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929732112613376 |
|---|---|
| author | Chan, Jun Shern Chowdhury, Neil Jaffe, Oliver Aung, James Sherburn, Dane Mays, Evan Starace, Giulio Liu, Kevin Maksin, Leon Patwardhan, Tejal Weng, Lilian Mądry, Aleksander |
| author_facet | Chan, Jun Shern Chowdhury, Neil Jaffe, Oliver Aung, James Sherburn, Dane Mays, Evan Starace, Giulio Liu, Kevin Maksin, Leon Patwardhan, Tejal Weng, Lilian Mądry, Aleksander |
| contents | We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering. To this end, we curate 75 ML engineering-related competitions from Kaggle, creating a diverse set of challenging tasks that test real-world ML engineering skills such as training models, preparing datasets, and running experiments. We establish human baselines for each competition using Kaggle's publicly available leaderboards. We use open-source agent scaffolds to evaluate several frontier language models on our benchmark, finding that the best-performing setup--OpenAI's o1-preview with AIDE scaffolding--achieves at least the level of a Kaggle bronze medal in 16.9% of competitions. In addition to our main results, we investigate various forms of resource scaling for AI agents and the impact of contamination from pre-training. We open-source our benchmark code (github.com/openai/mle-bench/) to facilitate future research in understanding the ML engineering capabilities of AI agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07095 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering Chan, Jun Shern Chowdhury, Neil Jaffe, Oliver Aung, James Sherburn, Dane Mays, Evan Starace, Giulio Liu, Kevin Maksin, Leon Patwardhan, Tejal Weng, Lilian Mądry, Aleksander Computation and Language We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering. To this end, we curate 75 ML engineering-related competitions from Kaggle, creating a diverse set of challenging tasks that test real-world ML engineering skills such as training models, preparing datasets, and running experiments. We establish human baselines for each competition using Kaggle's publicly available leaderboards. We use open-source agent scaffolds to evaluate several frontier language models on our benchmark, finding that the best-performing setup--OpenAI's o1-preview with AIDE scaffolding--achieves at least the level of a Kaggle bronze medal in 16.9% of competitions. In addition to our main results, we investigate various forms of resource scaling for AI agents and the impact of contamination from pre-training. We open-source our benchmark code (github.com/openai/mle-bench/) to facilitate future research in understanding the ML engineering capabilities of AI agents. |
| title | MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.07095 |