Environment-Aware Code Generation: How far are We?
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917209107857408 |
|---|---|
| author | Wu, Tongtong Chen, Rongyi Du, Wenjie Ma, Suyu Qi, Guilin Xing, Zhenchang Khadivi, Shahram Periyathambi, Ramesh Haffari, Gholamreza |
| author_facet | Wu, Tongtong Chen, Rongyi Du, Wenjie Ma, Suyu Qi, Guilin Xing, Zhenchang Khadivi, Shahram Periyathambi, Ramesh Haffari, Gholamreza |
| contents | Recent progress in large language models (LLMs) has improved code generation, but most evaluations still test isolated, small-scale code (e.g., a single function) under default or unspecified software environments. As a result, it is unclear whether LLMs can reliably generate executable code tailored to a user's specific environment. We present the first systematic study of Environment-Aware Code Generation (EACG), where generated code must be functionally correct and directly executable under arbitrary software configurations. To enable realistic evaluation, we introduce VersiBCB, a benchmark that is multi-package, execution-verified, and deprecation-aware, capturing complex and evolving environments that prior datasets often overlook. Using VersiBCB, we investigate three complementary adaptation axes: data, parameters, and cache, and develop representative strategies for each. Our results show that current LLMs struggle with environment-specific code generation, while our adaptations improve environment compatibility and executability. These findings highlight key challenges and opportunities for deploying LLMs in practical software engineering workflows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_12262 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Environment-Aware Code Generation: How far are We? Wu, Tongtong Chen, Rongyi Du, Wenjie Ma, Suyu Qi, Guilin Xing, Zhenchang Khadivi, Shahram Periyathambi, Ramesh Haffari, Gholamreza Software Engineering Computation and Language Recent progress in large language models (LLMs) has improved code generation, but most evaluations still test isolated, small-scale code (e.g., a single function) under default or unspecified software environments. As a result, it is unclear whether LLMs can reliably generate executable code tailored to a user's specific environment. We present the first systematic study of Environment-Aware Code Generation (EACG), where generated code must be functionally correct and directly executable under arbitrary software configurations. To enable realistic evaluation, we introduce VersiBCB, a benchmark that is multi-package, execution-verified, and deprecation-aware, capturing complex and evolving environments that prior datasets often overlook. Using VersiBCB, we investigate three complementary adaptation axes: data, parameters, and cache, and develop representative strategies for each. Our results show that current LLMs struggle with environment-specific code generation, while our adaptations improve environment compatibility and executability. These findings highlight key challenges and opportunities for deploying LLMs in practical software engineering workflows. |
| title | Environment-Aware Code Generation: How far are We? |
| topic | Software Engineering Computation and Language |
| url | https://arxiv.org/abs/2601.12262 |