Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910043070267392 |
|---|---|
| author | Wang, Kaixin Li, Tianlin Zhang, Xiaoyu Wang, Chong Sun, Weisong Liu, Yang Liu, Aishan Liu, Xianglong Shen, Chao Shi, Bin |
| author_facet | Wang, Kaixin Li, Tianlin Zhang, Xiaoyu Wang, Chong Sun, Weisong Liu, Yang Liu, Aishan Liu, Xianglong Shen, Chao Shi, Bin |
| contents | Code large language models (CodeLLMs) and agents are increasingly being integrated into complex software engineering tasks spanning the entire Software Development Life Cycle (SDLC). Benchmarking is critical for rigorously evaluating these capabilities. However, despite their growing significance, there remains a lack of comprehensive reviews that examine these benchmarks from an SDLC perspective. To bridge this gap, we propose a tiered analysis framework to systematically review 178 benchmarks from 461 papers, comprehensively characterizing them from the perspective of the SDLC. Our findings reveal a notable imbalance in the coverage of current benchmarks, with approximately 61\% focused on the software implementation phase in SDLC, while requirements engineering and software design phases receive minimal attention at only 5\% and 3\%, respectively. % Additionally, anti-contamination strategies are largely absent from current benchmarks, leading to an increased risk of data leakage. Furthermore, current benchmarks lack effective anti-contamination strategies, posing significant risks of data leakage and potentially inflated performance assessments. Finally, we identify key open challenges in current research and outline future directions to narrow the gap between the theoretical capabilities of CodeLLMs and agents and their practical effectiveness in real-world scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_05283 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents Wang, Kaixin Li, Tianlin Zhang, Xiaoyu Wang, Chong Sun, Weisong Liu, Yang Liu, Aishan Liu, Xianglong Shen, Chao Shi, Bin Software Engineering Artificial Intelligence Code large language models (CodeLLMs) and agents are increasingly being integrated into complex software engineering tasks spanning the entire Software Development Life Cycle (SDLC). Benchmarking is critical for rigorously evaluating these capabilities. However, despite their growing significance, there remains a lack of comprehensive reviews that examine these benchmarks from an SDLC perspective. To bridge this gap, we propose a tiered analysis framework to systematically review 178 benchmarks from 461 papers, comprehensively characterizing them from the perspective of the SDLC. Our findings reveal a notable imbalance in the coverage of current benchmarks, with approximately 61\% focused on the software implementation phase in SDLC, while requirements engineering and software design phases receive minimal attention at only 5\% and 3\%, respectively. % Additionally, anti-contamination strategies are largely absent from current benchmarks, leading to an increased risk of data leakage. Furthermore, current benchmarks lack effective anti-contamination strategies, posing significant risks of data leakage and potentially inflated performance assessments. Finally, we identify key open challenges in current research and outline future directions to narrow the gap between the theoretical capabilities of CodeLLMs and agents and their practical effectiveness in real-world scenarios. |
| title | Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2505.05283 |