Exploring the Effect of Multiple Natural Languages on Code Suggestion Using GitHub Copilot
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911769754075136 |
|---|---|
| author | Koyanagi, Kei Wang, Dong Noguchi, Kotaro Kondo, Masanari Serebrenik, Alexander Kamei, Yasutaka Ubayashi, Naoyasu |
| author_facet | Koyanagi, Kei Wang, Dong Noguchi, Kotaro Kondo, Masanari Serebrenik, Alexander Kamei, Yasutaka Ubayashi, Naoyasu |
| contents | GitHub Copilot is an AI-enabled tool that automates program synthesis. It has gained significant attention since its launch in 2021. Recent studies have extensively examined Copilot's capabilities in various programming tasks, as well as its security issues. However, little is known about the effect of different natural languages on code suggestion. Natural language is considered a social bias in the field of NLP, and this bias could impact the diversity of software engineering. To address this gap, we conducted an empirical study to investigate the effect of three popular natural languages (English, Japanese, and Chinese) on Copilot. We used 756 questions of varying difficulty levels from AtCoder contests for evaluation purposes. The results highlight that the capability varies across natural languages, with Chinese achieving the worst performance. Furthermore, regardless of the type of natural language, the performance decreases significantly as the difficulty of questions increases. Our work represents the initial step in comprehending the significance of natural languages in Copilot's capability and introduces promising opportunities for future endeavors. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_01438 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring the Effect of Multiple Natural Languages on Code Suggestion Using GitHub Copilot Koyanagi, Kei Wang, Dong Noguchi, Kotaro Kondo, Masanari Serebrenik, Alexander Kamei, Yasutaka Ubayashi, Naoyasu Software Engineering GitHub Copilot is an AI-enabled tool that automates program synthesis. It has gained significant attention since its launch in 2021. Recent studies have extensively examined Copilot's capabilities in various programming tasks, as well as its security issues. However, little is known about the effect of different natural languages on code suggestion. Natural language is considered a social bias in the field of NLP, and this bias could impact the diversity of software engineering. To address this gap, we conducted an empirical study to investigate the effect of three popular natural languages (English, Japanese, and Chinese) on Copilot. We used 756 questions of varying difficulty levels from AtCoder contests for evaluation purposes. The results highlight that the capability varies across natural languages, with Chinese achieving the worst performance. Furthermore, regardless of the type of natural language, the performance decreases significantly as the difficulty of questions increases. Our work represents the initial step in comprehending the significance of natural languages in Copilot's capability and introduces promising opportunities for future endeavors. |
| title | Exploring the Effect of Multiple Natural Languages on Code Suggestion Using GitHub Copilot |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2402.01438 |