Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917783509401600 |
|---|---|
| author | Cassano, Federico Li, Luisa Sethi, Akul Shinn, Noah Brennan-Jones, Abby Ginesin, Jacob Berman, Edward Chakhnashvili, George Lozhkov, Anton Anderson, Carolyn Jane Guha, Arjun |
| author_facet | Cassano, Federico Li, Luisa Sethi, Akul Shinn, Noah Brennan-Jones, Abby Ginesin, Jacob Berman, Edward Chakhnashvili, George Lozhkov, Anton Anderson, Carolyn Jane Guha, Arjun |
| contents | A significant amount of research is focused on developing and evaluating large language models for a variety of code synthesis tasks. These include synthesizing code from natural language, synthesizing tests from code, and synthesizing explanations of code. In contrast, the behavior of instructional code editing with LLMs is understudied. These are tasks in which the model is provided a block of code and an instruction to modify the code. The editing instruction may ask for a feature to be added or removed, describe a bug and ask for a fix, or ask for a different kind of solution. We introduce a carefully crafted benchmark of code editing tasks and use it to evaluate several cutting edge LLMs. Our evaluation exposes a significant gap between the capabilities of state-of-the-art open and closed models. For example, even GPT-3.5-Turbo is better than the best open model at code editing tasks. We also introduce a new, carefully curated, permissively licensed training dataset of code editing tasks coupled with natural language instructions. Using this training dataset, we show that we can fine-tune open Code LLMs to significantly improve their code editing capabilities, closing the gap between open and closed models. All code, data, and models are available at https://github.com/nuprl/CanItEdit. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_12450 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions Cassano, Federico Li, Luisa Sethi, Akul Shinn, Noah Brennan-Jones, Abby Ginesin, Jacob Berman, Edward Chakhnashvili, George Lozhkov, Anton Anderson, Carolyn Jane Guha, Arjun Software Engineering Artificial Intelligence Machine Learning Programming Languages A significant amount of research is focused on developing and evaluating large language models for a variety of code synthesis tasks. These include synthesizing code from natural language, synthesizing tests from code, and synthesizing explanations of code. In contrast, the behavior of instructional code editing with LLMs is understudied. These are tasks in which the model is provided a block of code and an instruction to modify the code. The editing instruction may ask for a feature to be added or removed, describe a bug and ask for a fix, or ask for a different kind of solution. We introduce a carefully crafted benchmark of code editing tasks and use it to evaluate several cutting edge LLMs. Our evaluation exposes a significant gap between the capabilities of state-of-the-art open and closed models. For example, even GPT-3.5-Turbo is better than the best open model at code editing tasks. We also introduce a new, carefully curated, permissively licensed training dataset of code editing tasks coupled with natural language instructions. Using this training dataset, we show that we can fine-tune open Code LLMs to significantly improve their code editing capabilities, closing the gap between open and closed models. All code, data, and models are available at https://github.com/nuprl/CanItEdit. |
| title | Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions |
| topic | Software Engineering Artificial Intelligence Machine Learning Programming Languages |
| url | https://arxiv.org/abs/2312.12450 |