ProgramBench: Can Language Models Rebuild Programs From Scratch?
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918484466728960 |
|---|---|
| author | Yang, John Lieret, Kilian Ma, Jeffrey Thakkar, Parth Pedchenko, Dmitrii Sootla, Sten McMilin, Emily Yin, Pengcheng Hou, Rui Synnaeve, Gabriel Yang, Diyi Press, Ofir |
| author_facet | Yang, John Lieret, Kilian Ma, Jeffrey Thakkar, Parth Pedchenko, Dmitrii Sootla, Sten McMilin, Emily Yin, Pengcheng Hou, Rui Synnaeve, Gabriel Yang, Diyi Press, Ofir |
| contents | Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95\% of tests on only 3\% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_03546 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ProgramBench: Can Language Models Rebuild Programs From Scratch? Yang, John Lieret, Kilian Ma, Jeffrey Thakkar, Parth Pedchenko, Dmitrii Sootla, Sten McMilin, Emily Yin, Pengcheng Hou, Rui Synnaeve, Gabriel Yang, Diyi Press, Ofir Software Engineering Artificial Intelligence Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95\% of tests on only 3\% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code. |
| title | ProgramBench: Can Language Models Rebuild Programs From Scratch? |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2605.03546 |