RepoST: Scalable Repository-Level Coding Environment Construction with Sandbox Testing
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917950016978944 |
|---|---|
| author | Xie, Yiqing Xie, Alex Sheth, Divyanshu Liu, Pengfei Fried, Daniel Rose, Carolyn |
| author_facet | Xie, Yiqing Xie, Alex Sheth, Divyanshu Liu, Pengfei Fried, Daniel Rose, Carolyn |
| contents | We present RepoST, a scalable method to construct environments that provide execution feedback for repository-level code generation for both training and evaluation. Unlike existing works that aim to build entire repositories for execution, which is challenging for both human and LLMs, we provide execution feedback with sandbox testing, which isolates a given target function and its dependencies to a separate script for testing. Sandbox testing reduces the complexity of external dependencies and enables constructing environments at a large scale. We use our method to construct RepoST-Train, a large-scale train set with 7,415 functions from 832 repositories. Training with the execution feedback provided by RepoST-Train leads to a performance gain of 5.5% Pass@1 on HumanEval and 3.5% Pass@1 on RepoEval. We also build an evaluation dataset, RepoST-Eval, and benchmark 12 code generation models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_07358 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RepoST: Scalable Repository-Level Coding Environment Construction with Sandbox Testing Xie, Yiqing Xie, Alex Sheth, Divyanshu Liu, Pengfei Fried, Daniel Rose, Carolyn Computation and Language Software Engineering We present RepoST, a scalable method to construct environments that provide execution feedback for repository-level code generation for both training and evaluation. Unlike existing works that aim to build entire repositories for execution, which is challenging for both human and LLMs, we provide execution feedback with sandbox testing, which isolates a given target function and its dependencies to a separate script for testing. Sandbox testing reduces the complexity of external dependencies and enables constructing environments at a large scale. We use our method to construct RepoST-Train, a large-scale train set with 7,415 functions from 832 repositories. Training with the execution feedback provided by RepoST-Train leads to a performance gain of 5.5% Pass@1 on HumanEval and 3.5% Pass@1 on RepoEval. We also build an evaluation dataset, RepoST-Eval, and benchmark 12 code generation models. |
| title | RepoST: Scalable Repository-Level Coding Environment Construction with Sandbox Testing |
| topic | Computation and Language Software Engineering |
| url | https://arxiv.org/abs/2503.07358 |