From Illusion to Insight: Change-Aware File-Level Software Defect Prediction Using Agentic AI
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911344908828672 |
|---|---|
| author | Hesamolhokama, Mohsen Rohani, Behnam Shafiee, Amirahmad Fazli, MohammadAmin Habibi, Jafar |
| author_facet | Hesamolhokama, Mohsen Rohani, Behnam Shafiee, Amirahmad Fazli, MohammadAmin Habibi, Jafar |
| contents | Much of the reported progress in file-level software defect prediction (SDP) is, in reality, nothing but an illusion of accuracy. Over the last decades, machine learning and deep learning models have reported increasing performance across software versions. However, since most files persist across releases and retain their defect labels, standard evaluation rewards label-persistence bias rather than reasoning about code changes. To address this issue, we reformulate SDP as a change-aware prediction task, in which models reason over code changes of a file within successive project versions, rather than relying on static file snapshots. Building on this formulation, we propose an LLM-driven, change-aware, multi-agent debate framework. Our experiments on multiple PROMISE projects show that traditional models achieve inflated F1, while failing on rare but critical defect-transition cases. In contrast, our change-aware reasoning and multi-agent debate framework yields more balanced performance across evolution subsets and significantly improves sensitivity to defect introductions. These results highlight fundamental flaws in current SDP evaluation practices and emphasize the need for change-aware reasoning in practical defect prediction. The source code is publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_23875 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Illusion to Insight: Change-Aware File-Level Software Defect Prediction Using Agentic AI Hesamolhokama, Mohsen Rohani, Behnam Shafiee, Amirahmad Fazli, MohammadAmin Habibi, Jafar Software Engineering Much of the reported progress in file-level software defect prediction (SDP) is, in reality, nothing but an illusion of accuracy. Over the last decades, machine learning and deep learning models have reported increasing performance across software versions. However, since most files persist across releases and retain their defect labels, standard evaluation rewards label-persistence bias rather than reasoning about code changes. To address this issue, we reformulate SDP as a change-aware prediction task, in which models reason over code changes of a file within successive project versions, rather than relying on static file snapshots. Building on this formulation, we propose an LLM-driven, change-aware, multi-agent debate framework. Our experiments on multiple PROMISE projects show that traditional models achieve inflated F1, while failing on rare but critical defect-transition cases. In contrast, our change-aware reasoning and multi-agent debate framework yields more balanced performance across evolution subsets and significantly improves sensitivity to defect introductions. These results highlight fundamental flaws in current SDP evaluation practices and emphasize the need for change-aware reasoning in practical defect prediction. The source code is publicly available. |
| title | From Illusion to Insight: Change-Aware File-Level Software Defect Prediction Using Agentic AI |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2512.23875 |