MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908360750661632 |
|---|---|
| author | Qiang, Rushi Zhuang, Yuchen Li, Yinghao K, Dingu Sagar V Zhang, Rongzhi Li, Changhao Wong, Ian Shu-Hei Yang, Sherry Liang, Percy Zhang, Chao Dai, Bo |
| author_facet | Qiang, Rushi Zhuang, Yuchen Li, Yinghao K, Dingu Sagar V Zhang, Rongzhi Li, Changhao Wong, Ian Shu-Hei Yang, Sherry Liang, Percy Zhang, Chao Dai, Bo |
| contents | We introduce MLE-Dojo, a Gym-style framework for systematically reinforcement learning, evaluating, and improving autonomous large language model (LLM) agents in iterative machine learning engineering (MLE) workflows. Unlike existing benchmarks that primarily rely on static datasets or single-attempt evaluations, MLE-Dojo provides an interactive environment enabling agents to iteratively experiment, debug, and refine solutions through structured feedback loops. Built upon 200+ real-world Kaggle challenges, MLE-Dojo covers diverse, open-ended MLE tasks carefully curated to reflect realistic engineering scenarios such as data processing, architecture search, hyperparameter tuning, and code debugging. Its fully executable environment supports comprehensive agent training via both supervised fine-tuning and reinforcement learning, facilitating iterative experimentation, realistic data sampling, and real-time outcome verification. Extensive evaluations of eight frontier LLMs reveal that while current models achieve meaningful iterative improvements, they still exhibit significant limitations in autonomously generating long-horizon solutions and efficiently resolving complex errors. Furthermore, MLE-Dojo's flexible and extensible architecture seamlessly integrates diverse data sources, tools, and evaluation protocols, uniquely enabling model-based agent tuning and promoting interoperability, scalability, and reproducibility. We open-source our framework and benchmarks to foster community-driven innovation towards next-generation MLE agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_07782 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering Qiang, Rushi Zhuang, Yuchen Li, Yinghao K, Dingu Sagar V Zhang, Rongzhi Li, Changhao Wong, Ian Shu-Hei Yang, Sherry Liang, Percy Zhang, Chao Dai, Bo Machine Learning We introduce MLE-Dojo, a Gym-style framework for systematically reinforcement learning, evaluating, and improving autonomous large language model (LLM) agents in iterative machine learning engineering (MLE) workflows. Unlike existing benchmarks that primarily rely on static datasets or single-attempt evaluations, MLE-Dojo provides an interactive environment enabling agents to iteratively experiment, debug, and refine solutions through structured feedback loops. Built upon 200+ real-world Kaggle challenges, MLE-Dojo covers diverse, open-ended MLE tasks carefully curated to reflect realistic engineering scenarios such as data processing, architecture search, hyperparameter tuning, and code debugging. Its fully executable environment supports comprehensive agent training via both supervised fine-tuning and reinforcement learning, facilitating iterative experimentation, realistic data sampling, and real-time outcome verification. Extensive evaluations of eight frontier LLMs reveal that while current models achieve meaningful iterative improvements, they still exhibit significant limitations in autonomously generating long-horizon solutions and efficiently resolving complex errors. Furthermore, MLE-Dojo's flexible and extensible architecture seamlessly integrates diverse data sources, tools, and evaluation protocols, uniquely enabling model-based agent tuning and promoting interoperability, scalability, and reproducibility. We open-source our framework and benchmarks to foster community-driven innovation towards next-generation MLE agents. |
| title | MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.07782 |