Examination of Code generated by Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910582143188992 |
|---|---|
| author | Beer, Robin Feix, Alexander Guttzeit, Tim Muras, Tamara Müller, Vincent Rauscher, Maurice Schäffler, Florian Löwe, Welf |
| author_facet | Beer, Robin Feix, Alexander Guttzeit, Tim Muras, Tamara Müller, Vincent Rauscher, Maurice Schäffler, Florian Löwe, Welf |
| contents | Large language models (LLMs), such as ChatGPT and Copilot, are transforming software development by automating code generation and, arguably, enable rapid prototyping, support education, and boost productivity. Therefore, correctness and quality of the generated code should be on par with manually written code. To assess the current state of LLMs in generating correct code of high quality, we conducted controlled experiments with ChatGPT and Copilot: we let the LLMs generate simple algorithms in Java and Python along with the corresponding unit tests and assessed the correctness and the quality (coverage) of the generated (test) codes. We observed significant differences between the LLMs, between the languages, between algorithm and test codes, and over time. The present paper reports these results together with the experimental methods allowing repeated and comparable assessments for more algorithms, languages, and LLMs over time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_16601 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Examination of Code generated by Large Language Models Beer, Robin Feix, Alexander Guttzeit, Tim Muras, Tamara Müller, Vincent Rauscher, Maurice Schäffler, Florian Löwe, Welf Software Engineering Artificial Intelligence I.2.2 Large language models (LLMs), such as ChatGPT and Copilot, are transforming software development by automating code generation and, arguably, enable rapid prototyping, support education, and boost productivity. Therefore, correctness and quality of the generated code should be on par with manually written code. To assess the current state of LLMs in generating correct code of high quality, we conducted controlled experiments with ChatGPT and Copilot: we let the LLMs generate simple algorithms in Java and Python along with the corresponding unit tests and assessed the correctness and the quality (coverage) of the generated (test) codes. We observed significant differences between the LLMs, between the languages, between algorithm and test codes, and over time. The present paper reports these results together with the experimental methods allowing repeated and comparable assessments for more algorithms, languages, and LLMs over time. |
| title | Examination of Code generated by Large Language Models |
| topic | Software Engineering Artificial Intelligence I.2.2 |
| url | https://arxiv.org/abs/2408.16601 |