How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929681496801280 |
|---|---|
| author | Velasco, Alejandro Rodriguez-Cardenas, Daniel Alif, Luftar Rahman Palacio, David N. Poshyvanyk, Denys |
| author_facet | Velasco, Alejandro Rodriguez-Cardenas, Daniel Alif, Luftar Rahman Palacio, David N. Poshyvanyk, Denys |
| contents | Large Language Models (LLMs) have shown significant potential in automating software engineering tasks, particularly in code generation. However, current evaluation benchmarks, which primarily focus on accuracy, fall short in assessing the quality of the code generated by these models, specifically their tendency to produce code smells. To address this limitation, we introduce CodeSmellEval, a benchmark designed to evaluate the propensity of LLMs for generating code smells. Our benchmark includes a novel metric: Propensity Smelly Score (PSC), and a curated dataset of method-level code smells: CodeSmellData. To demonstrate the use of CodeSmellEval, we conducted a case study with two state-of-the-art LLMs, CodeLlama and Mistral. The results reveal that both models tend to generate code smells, such as simplifiable-condition and consider-merging-isinstance. These findings highlight the effectiveness of our benchmark in evaluating LLMs, providing valuable insights into their reliability and their propensity to introduce code smells in code generation tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_18989 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study Velasco, Alejandro Rodriguez-Cardenas, Daniel Alif, Luftar Rahman Palacio, David N. Poshyvanyk, Denys Software Engineering Artificial Intelligence Large Language Models (LLMs) have shown significant potential in automating software engineering tasks, particularly in code generation. However, current evaluation benchmarks, which primarily focus on accuracy, fall short in assessing the quality of the code generated by these models, specifically their tendency to produce code smells. To address this limitation, we introduce CodeSmellEval, a benchmark designed to evaluate the propensity of LLMs for generating code smells. Our benchmark includes a novel metric: Propensity Smelly Score (PSC), and a curated dataset of method-level code smells: CodeSmellData. To demonstrate the use of CodeSmellEval, we conducted a case study with two state-of-the-art LLMs, CodeLlama and Mistral. The results reveal that both models tend to generate code smells, such as simplifiable-condition and consider-merging-isinstance. These findings highlight the effectiveness of our benchmark in evaluating LLMs, providing valuable insights into their reliability and their propensity to introduce code smells in code generation tasks. |
| title | How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2412.18989 |