Investigating Test Overfitting on SWE-bench
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918426726891520 |
|---|---|
| author | Ahmed, Toufique Ganhotra, Jatin Shinnar, Avraham Hirzel, Martin |
| author_facet | Ahmed, Toufique Ganhotra, Jatin Shinnar, Avraham Hirzel, Martin |
| contents | Tests can be useful towards resolving issues on code repositories. However, relying too much on tests for issue resolution can lead to code that technically passes observed tests but actually misses important cases or even breaks functionality. This problem, called test overfitting, is exacerbated by the fact that issues usually lack readily executable tests. Instead, several issue resolution systems use tests auto-generated from issues, which may be imperfect. Some systems even iteratively refine code and tests jointly. This paper presents the first empirical study of test overfitting in this setting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_16858 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Investigating Test Overfitting on SWE-bench Ahmed, Toufique Ganhotra, Jatin Shinnar, Avraham Hirzel, Martin Software Engineering Machine Learning Tests can be useful towards resolving issues on code repositories. However, relying too much on tests for issue resolution can lead to code that technically passes observed tests but actually misses important cases or even breaks functionality. This problem, called test overfitting, is exacerbated by the fact that issues usually lack readily executable tests. Instead, several issue resolution systems use tests auto-generated from issues, which may be imperfect. Some systems even iteratively refine code and tests jointly. This paper presents the first empirical study of test overfitting in this setting. |
| title | Investigating Test Overfitting on SWE-bench |
| topic | Software Engineering Machine Learning |
| url | https://arxiv.org/abs/2511.16858 |