Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914274206547968 |
|---|---|
| author | Ahmed, Toufique Ganhotra, Jatin Shinnar, Avraham Hirzel, Martin |
| author_facet | Ahmed, Toufique Ganhotra, Jatin Shinnar, Avraham Hirzel, Martin |
| contents | A software engineering issue (SWE issue) is easier to resolve when accompanied by a reproduction test. Unfortunately, most issues do not come with functioning reproduction tests, so this paper explores how to generate them automatically. The primary challenge in this setting is that the code to be tested is either missing or wrong, as evidenced by the existence of the issue in the first place. This has held back test generation for this setting: without the correct code to execute, it is difficult to leverage execution feedback to generate good tests. This paper introduces novel techniques for leveraging execution feedback to get around this problem, implemented in a new reproduction test generator called e-Otter++. Experiments show that e-Otter++ represents a leap ahead in the state-of-the-art for this problem, generating tests with an average fail-to-pass rate of 63% on the TDD-Bench Verified benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_06365 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection Ahmed, Toufique Ganhotra, Jatin Shinnar, Avraham Hirzel, Martin Software Engineering A software engineering issue (SWE issue) is easier to resolve when accompanied by a reproduction test. Unfortunately, most issues do not come with functioning reproduction tests, so this paper explores how to generate them automatically. The primary challenge in this setting is that the code to be tested is either missing or wrong, as evidenced by the existence of the issue in the first place. This has held back test generation for this setting: without the correct code to execute, it is difficult to leverage execution feedback to generate good tests. This paper introduces novel techniques for leveraging execution feedback to get around this problem, implemented in a new reproduction test generator called e-Otter++. Experiments show that e-Otter++ represents a leap ahead in the state-of-the-art for this problem, generating tests with an average fail-to-pass rate of 63% on the TDD-Bench Verified benchmark. |
| title | Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2508.06365 |