Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915760254746624 |
|---|---|
| author | Cambronero, José Tufano, Michele Shi, Sherry Wei, Renyao Uy, Grant Cheng, Runxiang Liu, Chin-Jung Pan, Shiying Chandra, Satish Rondon, Pat |
| author_facet | Cambronero, José Tufano, Michele Shi, Sherry Wei, Renyao Uy, Grant Cheng, Runxiang Liu, Chin-Jung Pan, Shiying Chandra, Satish Rondon, Pat |
| contents | Agentic Automated Program Repair (APR) is increasingly tackling complex, repository-level bugs in industry, but ultimately these patches still need to be reviewed by a human before committing them to ensure they address the bug. Showing patches unlikely to be accepted can lead to substantial noise, wasting valuable developer time and eroding trust in automated code changes. We introduce two complementary LLM-based policies to reduce such noise: bug abstention and patch validation policies. Bug abstention excludes bugs that the agentic APR system is unlikely to fix. Patch validation rejects patches that are unlikely to be a good fix for the given bug. We evaluate both policies on three sets of bugs from Google's codebase, and their candidate patches generated by an internal agentic APR system. On a set of 174 human-reported bugs, removing bugs and patches rejected by our policies can raise success rates by up to 13 percentage points and 15 percentage points, respectively, and by up to 39 percentage points in combination. On null pointer exceptions and sanitizer-reported bugs with machine-generated bug reports, patch validation also improves average single-sample success rates. This two-policy approach provides a practical path to the reliable, industrial-scale deployment of agentic APR systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_03217 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair Cambronero, José Tufano, Michele Shi, Sherry Wei, Renyao Uy, Grant Cheng, Runxiang Liu, Chin-Jung Pan, Shiying Chandra, Satish Rondon, Pat Software Engineering Artificial Intelligence D.2.5; I.2.2 Agentic Automated Program Repair (APR) is increasingly tackling complex, repository-level bugs in industry, but ultimately these patches still need to be reviewed by a human before committing them to ensure they address the bug. Showing patches unlikely to be accepted can lead to substantial noise, wasting valuable developer time and eroding trust in automated code changes. We introduce two complementary LLM-based policies to reduce such noise: bug abstention and patch validation policies. Bug abstention excludes bugs that the agentic APR system is unlikely to fix. Patch validation rejects patches that are unlikely to be a good fix for the given bug. We evaluate both policies on three sets of bugs from Google's codebase, and their candidate patches generated by an internal agentic APR system. On a set of 174 human-reported bugs, removing bugs and patches rejected by our policies can raise success rates by up to 13 percentage points and 15 percentage points, respectively, and by up to 39 percentage points in combination. On null pointer exceptions and sanitizer-reported bugs with machine-generated bug reports, patch validation also improves average single-sample success rates. This two-policy approach provides a practical path to the reliable, industrial-scale deployment of agentic APR systems. |
| title | Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair |
| topic | Software Engineering Artificial Intelligence D.2.5; I.2.2 |
| url | https://arxiv.org/abs/2510.03217 |