VersiCode: Towards Version-controllable Code Generation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917804820660224 |
|---|---|
| author | Wu, Tongtong Wu, Weigang Wang, Xingyu Xu, Kang Ma, Suyu Jiang, Bo Yang, Ping Xing, Zhenchang Li, Yuan-Fang Haffari, Gholamreza |
| author_facet | Wu, Tongtong Wu, Weigang Wang, Xingyu Xu, Kang Ma, Suyu Jiang, Bo Yang, Ping Xing, Zhenchang Li, Yuan-Fang Haffari, Gholamreza |
| contents | Large Language Models (LLMs) have made tremendous strides in code generation, but existing research fails to account for the dynamic nature of software development, marked by frequent library updates. This gap significantly limits LLMs' deployment in realistic settings. In this paper, we propose two novel tasks aimed at bridging this gap: version-specific code completion (VSCC) and version-aware code migration (VACM). In conjunction, we introduce VersiCode, a comprehensive Python dataset specifically designed to evaluate LLMs on these two tasks, together with a novel evaluation metric, Critical Diff Check (CDC@1), which assesses code generation against evolving API requirements. We conduct an extensive evaluation on VersiCode, which reveals that version-controllable code generation is indeed a significant challenge, even for GPT-4o and other strong frontier models. We believe the novel tasks, dataset, and metric open up a new, important research direction that will further enhance LLMs' real-world applicability. The code and resources can be found at https://github.com/wutong8023/VersiCode. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07411 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VersiCode: Towards Version-controllable Code Generation Wu, Tongtong Wu, Weigang Wang, Xingyu Xu, Kang Ma, Suyu Jiang, Bo Yang, Ping Xing, Zhenchang Li, Yuan-Fang Haffari, Gholamreza Software Engineering Computation and Language Large Language Models (LLMs) have made tremendous strides in code generation, but existing research fails to account for the dynamic nature of software development, marked by frequent library updates. This gap significantly limits LLMs' deployment in realistic settings. In this paper, we propose two novel tasks aimed at bridging this gap: version-specific code completion (VSCC) and version-aware code migration (VACM). In conjunction, we introduce VersiCode, a comprehensive Python dataset specifically designed to evaluate LLMs on these two tasks, together with a novel evaluation metric, Critical Diff Check (CDC@1), which assesses code generation against evolving API requirements. We conduct an extensive evaluation on VersiCode, which reveals that version-controllable code generation is indeed a significant challenge, even for GPT-4o and other strong frontier models. We believe the novel tasks, dataset, and metric open up a new, important research direction that will further enhance LLMs' real-world applicability. The code and resources can be found at https://github.com/wutong8023/VersiCode. |
| title | VersiCode: Towards Version-controllable Code Generation |
| topic | Software Engineering Computation and Language |
| url | https://arxiv.org/abs/2406.07411 |