Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.17238 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929554464964608 |
|---|---|
| author | Chi, Yizhou Lin, Yizhang Hong, Sirui Pan, Duyi Fei, Yaying Mei, Guanghao Liu, Bangbang Pang, Tianqi Kwok, Jacky Zhang, Ceyao Liu, Bang Wu, Chenglin |
| author_facet | Chi, Yizhou Lin, Yizhang Hong, Sirui Pan, Duyi Fei, Yaying Mei, Guanghao Liu, Bangbang Pang, Tianqi Kwok, Jacky Zhang, Ceyao Liu, Bang Wu, Chenglin |
| contents | Automated Machine Learning (AutoML) approaches encompass traditional methods that optimize fixed pipelines for model selection and ensembling, as well as newer LLM-based frameworks that autonomously build pipelines. While LLM-based agents have shown promise in automating machine learning tasks, they often generate low-diversity and suboptimal code, even after multiple iterations. To overcome these limitations, we introduce Tree-Search Enhanced LLM Agents (SELA), an innovative agent-based system that leverages Monte Carlo Tree Search (MCTS) to optimize the AutoML process. By representing pipeline configurations as trees, our framework enables agents to conduct experiments intelligently and iteratively refine their strategies, facilitating a more effective exploration of the machine learning solution space. This novel approach allows SELA to discover optimal pathways based on experimental feedback, improving the overall quality of the solutions. In an extensive evaluation across 20 machine learning datasets, we compare the performance of traditional and agent-based AutoML methods, demonstrating that SELA achieves a win rate of 65% to 80% against each baseline across all datasets. These results underscore the significant potential of agent-based strategies in AutoML, offering a fresh perspective on tackling complex machine learning challenges. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_17238 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning Chi, Yizhou Lin, Yizhang Hong, Sirui Pan, Duyi Fei, Yaying Mei, Guanghao Liu, Bangbang Pang, Tianqi Kwok, Jacky Zhang, Ceyao Liu, Bang Wu, Chenglin Artificial Intelligence Computation and Language Machine Learning Software Engineering Automated Machine Learning (AutoML) approaches encompass traditional methods that optimize fixed pipelines for model selection and ensembling, as well as newer LLM-based frameworks that autonomously build pipelines. While LLM-based agents have shown promise in automating machine learning tasks, they often generate low-diversity and suboptimal code, even after multiple iterations. To overcome these limitations, we introduce Tree-Search Enhanced LLM Agents (SELA), an innovative agent-based system that leverages Monte Carlo Tree Search (MCTS) to optimize the AutoML process. By representing pipeline configurations as trees, our framework enables agents to conduct experiments intelligently and iteratively refine their strategies, facilitating a more effective exploration of the machine learning solution space. This novel approach allows SELA to discover optimal pathways based on experimental feedback, improving the overall quality of the solutions. In an extensive evaluation across 20 machine learning datasets, we compare the performance of traditional and agent-based AutoML methods, demonstrating that SELA achieves a win rate of 65% to 80% against each baseline across all datasets. These results underscore the significant potential of agent-based strategies in AutoML, offering a fresh perspective on tackling complex machine learning challenges. |
| title | SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning |
| topic | Artificial Intelligence Computation and Language Machine Learning Software Engineering |
| url | https://arxiv.org/abs/2410.17238 |