Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914911543623680 |
|---|---|
| author | Zhang, Kexun Yao, Weiran Liu, Zuxin Feng, Yihao Liu, Zhiwei Murthy, Rithesh Lan, Tian Li, Lei Lou, Renze Xu, Jiacheng Pang, Bo Zhou, Yingbo Heinecke, Shelby Savarese, Silvio Wang, Huan Xiong, Caiming |
| author_facet | Zhang, Kexun Yao, Weiran Liu, Zuxin Feng, Yihao Liu, Zhiwei Murthy, Rithesh Lan, Tian Li, Lei Lou, Renze Xu, Jiacheng Pang, Bo Zhou, Yingbo Heinecke, Shelby Savarese, Silvio Wang, Huan Xiong, Caiming |
| contents | Large language model (LLM) agents have shown great potential in solving real-world software engineering (SWE) problems. The most advanced open-source SWE agent can resolve over 27% of real GitHub issues in SWE-Bench Lite. However, these sophisticated agent frameworks exhibit varying strengths, excelling in certain tasks while underperforming in others. To fully harness the diversity of these agents, we propose DEI (Diversity Empowered Intelligence), a framework that leverages their unique expertise. DEI functions as a meta-module atop existing SWE agent frameworks, managing agent collectives for enhanced problem-solving. Experimental results show that a DEI-guided committee of agents is able to surpass the best individual agent's performance by a large margin. For instance, a group of open-source SWE agents, with a maximum individual resolve rate of 27.3% on SWE-Bench Lite, can achieve a 34.3% resolve rate with DEI, making a 25% improvement and beating most closed-source solutions. Our best-performing group excels with a 55% resolve rate, securing the highest ranking on SWE-Bench Lite. Our findings contribute to the growing body of research on collaborative AI systems and their potential to solve complex software engineering challenges. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_07060 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents Zhang, Kexun Yao, Weiran Liu, Zuxin Feng, Yihao Liu, Zhiwei Murthy, Rithesh Lan, Tian Li, Lei Lou, Renze Xu, Jiacheng Pang, Bo Zhou, Yingbo Heinecke, Shelby Savarese, Silvio Wang, Huan Xiong, Caiming Software Engineering Artificial Intelligence Computation and Language Machine Learning Large language model (LLM) agents have shown great potential in solving real-world software engineering (SWE) problems. The most advanced open-source SWE agent can resolve over 27% of real GitHub issues in SWE-Bench Lite. However, these sophisticated agent frameworks exhibit varying strengths, excelling in certain tasks while underperforming in others. To fully harness the diversity of these agents, we propose DEI (Diversity Empowered Intelligence), a framework that leverages their unique expertise. DEI functions as a meta-module atop existing SWE agent frameworks, managing agent collectives for enhanced problem-solving. Experimental results show that a DEI-guided committee of agents is able to surpass the best individual agent's performance by a large margin. For instance, a group of open-source SWE agents, with a maximum individual resolve rate of 27.3% on SWE-Bench Lite, can achieve a 34.3% resolve rate with DEI, making a 25% improvement and beating most closed-source solutions. Our best-performing group excels with a 55% resolve rate, securing the highest ranking on SWE-Bench Lite. Our findings contribute to the growing body of research on collaborative AI systems and their potential to solve complex software engineering challenges. |
| title | Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents |
| topic | Software Engineering Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2408.07060 |