Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912690669092864 |
|---|---|
| author | Sharifloo, Amir Molzam Heydari, Maedeh Kazerooni, Parsa Maninger, Daniel Mezini, Mira |
| author_facet | Sharifloo, Amir Molzam Heydari, Maedeh Kazerooni, Parsa Maninger, Daniel Mezini, Mira |
| contents | Large Language Models (LLMs) have achieved remarkable success in code generation, and the race to improve their performance has become a central focus of AI research. Benchmarks and leaderboards are increasingly popular, offering quantitative rankings of LLMs. However, they provide limited insight into the tasks that LLMs consistently fail to solve - information that is crucial for understanding current limitations and guiding the development of more capable models. To address this gap, we examined code generation tasks across four popular benchmarks, identifying those that major LLMs are most likely to fail. To understand the causes of these failures, we investigated whether the static complexity of solution code contributes to them, followed by a systematic inspection of 114 tasks that LLMs consistently struggled with. Our analysis revealed four recurring patterns of weaknesses in LLMs, as well as common complications within benchmark tasks that most often lead to failure. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_04355 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks Sharifloo, Amir Molzam Heydari, Maedeh Kazerooni, Parsa Maninger, Daniel Mezini, Mira Software Engineering Machine Learning Large Language Models (LLMs) have achieved remarkable success in code generation, and the race to improve their performance has become a central focus of AI research. Benchmarks and leaderboards are increasingly popular, offering quantitative rankings of LLMs. However, they provide limited insight into the tasks that LLMs consistently fail to solve - information that is crucial for understanding current limitations and guiding the development of more capable models. To address this gap, we examined code generation tasks across four popular benchmarks, identifying those that major LLMs are most likely to fail. To understand the causes of these failures, we investigated whether the static complexity of solution code contributes to them, followed by a systematic inspection of 114 tasks that LLMs consistently struggled with. Our analysis revealed four recurring patterns of weaknesses in LLMs, as well as common complications within benchmark tasks that most often lead to failure. |
| title | Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks |
| topic | Software Engineering Machine Learning |
| url | https://arxiv.org/abs/2511.04355 |