RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908524045402112 |
|---|---|
| author | Liu, Jingjing Liu, Zeming Cheng, Zihao He, Mengliang Shi, Xiaoming Guo, Yuhang Zhu, Xiangrong Guo, Yuanfang Wang, Yunhong Wang, Haifeng |
| author_facet | Liu, Jingjing Liu, Zeming Cheng, Zihao He, Mengliang Shi, Xiaoming Guo, Yuhang Zhu, Xiangrong Guo, Yuanfang Wang, Yunhong Wang, Haifeng |
| contents | Large Language Models (LLMs) have exhibited significant proficiency in code debugging, especially in automatic program repair, which may substantially reduce the time consumption of developers and enhance their efficiency. Significant advancements in debugging datasets have been made to promote the development of code debugging. However, these datasets primarily focus on assessing the LLM's function-level code repair capabilities, neglecting the more complex and realistic repository-level scenarios, which leads to an incomplete understanding of the LLM's challenges in repository-level debugging. While several repository-level datasets have been proposed, they often suffer from limitations such as limited diversity of tasks, languages, and error types. To mitigate this challenge, this paper introduces RepoDebug, a multi-task and multi-language repository-level code debugging dataset with 22 subtypes of errors that supports 8 commonly used programming languages and 3 debugging tasks. Furthermore, we conduct evaluation experiments on 10 LLMs, where Claude 3.5 Sonnect, the best-performing model, still cannot perform well in repository-level debugging. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_04078 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models Liu, Jingjing Liu, Zeming Cheng, Zihao He, Mengliang Shi, Xiaoming Guo, Yuhang Zhu, Xiangrong Guo, Yuanfang Wang, Yunhong Wang, Haifeng Software Engineering Artificial Intelligence Large Language Models (LLMs) have exhibited significant proficiency in code debugging, especially in automatic program repair, which may substantially reduce the time consumption of developers and enhance their efficiency. Significant advancements in debugging datasets have been made to promote the development of code debugging. However, these datasets primarily focus on assessing the LLM's function-level code repair capabilities, neglecting the more complex and realistic repository-level scenarios, which leads to an incomplete understanding of the LLM's challenges in repository-level debugging. While several repository-level datasets have been proposed, they often suffer from limitations such as limited diversity of tasks, languages, and error types. To mitigate this challenge, this paper introduces RepoDebug, a multi-task and multi-language repository-level code debugging dataset with 22 subtypes of errors that supports 8 commonly used programming languages and 3 debugging tasks. Furthermore, we conduct evaluation experiments on 10 LLMs, where Claude 3.5 Sonnect, the best-performing model, still cannot perform well in repository-level debugging. |
| title | RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2509.04078 |