Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.14419 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915452219817984 |
|---|---|
| author | Blyth, Scott Licorish, Sherlock A. Treude, Christoph Wagner, Markus |
| author_facet | Blyth, Scott Licorish, Sherlock A. Treude, Christoph Wagner, Markus |
| contents | Large language models (LLMs) have demonstrated impressive capabilities in code generation, achieving high scores on benchmarks such as HumanEval and MBPP. However, these benchmarks primarily assess functional correctness and neglect broader dimensions of code quality, including security, reliability, readability, and maintainability. In this work, we systematically evaluate the ability of LLMs to generate high-quality code across multiple dimensions using the PythonSecurityEval benchmark. We introduce an iterative static analysis-driven prompting algorithm that leverages Bandit and Pylint to identify and resolve code quality issues. Our experiments with GPT-4o show substantial improvements: security issues reduced from >40% to 13%, readability violations from >80% to 11%, and reliability warnings from >50% to 11% within ten iterations. These results demonstrate that LLMs, when guided by static analysis feedback, can significantly enhance code quality beyond functional correctness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_14419 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Static Analysis as a Feedback Loop: Enhancing LLM-Generated Code Beyond Correctness Blyth, Scott Licorish, Sherlock A. Treude, Christoph Wagner, Markus Software Engineering Large language models (LLMs) have demonstrated impressive capabilities in code generation, achieving high scores on benchmarks such as HumanEval and MBPP. However, these benchmarks primarily assess functional correctness and neglect broader dimensions of code quality, including security, reliability, readability, and maintainability. In this work, we systematically evaluate the ability of LLMs to generate high-quality code across multiple dimensions using the PythonSecurityEval benchmark. We introduce an iterative static analysis-driven prompting algorithm that leverages Bandit and Pylint to identify and resolve code quality issues. Our experiments with GPT-4o show substantial improvements: security issues reduced from >40% to 13%, readability violations from >80% to 11%, and reliability warnings from >50% to 11% within ten iterations. These results demonstrate that LLMs, when guided by static analysis feedback, can significantly enhance code quality beyond functional correctness. |
| title | Static Analysis as a Feedback Loop: Enhancing LLM-Generated Code Beyond Correctness |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2508.14419 |