Syntax Is Not Enough: An Empirical Study of Small Transformer Models for Neural Code Repair
Fuente:
arXiv
Guardado en:
| Autor principal: | |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917171782746112 |
|---|---|
| author | Samant, Shaunak |
| author_facet | Samant, Shaunak |
| contents | Automated program repair using neural models has shown promising results on benchmark datasets, yet practical deployment remains limited. In this study, we examine whether a small transformer model can meaningfully repair real-world Java bugs and whether syntactic correctness is a reliable proxy for semantic correctness.
We fine-tune CodeT5-small (60.5M parameters) on 52,364 Java bug-fix pairs from CodeXGLUE and evaluate both token-level performance and syntactic validity using AST parsing. While the model converges cleanly and achieves high grammatical correctness, producing syntactically valid Java code in approximately ninety-four percent of cases, it fails to generate correct repairs under exact-match evaluation, achieving zero exact matches. In approximately eighty percent of cases, the model reproduces the buggy input verbatim. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_22216 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Syntax Is Not Enough: An Empirical Study of Small Transformer Models for Neural Code Repair Samant, Shaunak Software Engineering Automated program repair using neural models has shown promising results on benchmark datasets, yet practical deployment remains limited. In this study, we examine whether a small transformer model can meaningfully repair real-world Java bugs and whether syntactic correctness is a reliable proxy for semantic correctness. We fine-tune CodeT5-small (60.5M parameters) on 52,364 Java bug-fix pairs from CodeXGLUE and evaluate both token-level performance and syntactic validity using AST parsing. While the model converges cleanly and achieves high grammatical correctness, producing syntactically valid Java code in approximately ninety-four percent of cases, it fails to generate correct repairs under exact-match evaluation, achieving zero exact matches. In approximately eighty percent of cases, the model reproduces the buggy input verbatim. |
| title | Syntax Is Not Enough: An Empirical Study of Small Transformer Models for Neural Code Repair |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2512.22216 |