How Natural Language Proficiency Shapes GenAI Code for Software Engineering Tasks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911251194445824 |
|---|---|
| author | Rojpaisarnkit, Ruksit Fan, Youmei Matsumoto, Kenichi Kula, Raula Gaikovina |
| author_facet | Rojpaisarnkit, Ruksit Fan, Youmei Matsumoto, Kenichi Kula, Raula Gaikovina |
| contents | With the widespread adoption of Foundation Model (FM)-powered tools in software engineering, the natural language prompt has become a critical interface between developers and Large Language Models (LLMs). While much research has focused on prompt structure, the natural language proficiency is an underexplored factor that can influence the quality of generated code. This paper investigates whether the English language proficiency itself independent of the prompting technique affects the proficiency and correctness of code generated by LLMs. Using the HumanEval dataset, we systematically varied the English proficiency of prompts from basic to advanced for 164 programming tasks and measured the resulting code proficiency and correctness. Our findings show that LLMs default to an intermediate (B2) natural language level. While the effect on the resulting code proficiency was model-dependent, we found that higher-proficiency prompts consistently yielded more correct code across all models. These results demonstrate that natural language proficiency is a key lever for controlling code generation, helping developers tailor AI output and improve the reliability of solutions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_04115 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | How Natural Language Proficiency Shapes GenAI Code for Software Engineering Tasks Rojpaisarnkit, Ruksit Fan, Youmei Matsumoto, Kenichi Kula, Raula Gaikovina Software Engineering Programming Languages With the widespread adoption of Foundation Model (FM)-powered tools in software engineering, the natural language prompt has become a critical interface between developers and Large Language Models (LLMs). While much research has focused on prompt structure, the natural language proficiency is an underexplored factor that can influence the quality of generated code. This paper investigates whether the English language proficiency itself independent of the prompting technique affects the proficiency and correctness of code generated by LLMs. Using the HumanEval dataset, we systematically varied the English proficiency of prompts from basic to advanced for 164 programming tasks and measured the resulting code proficiency and correctness. Our findings show that LLMs default to an intermediate (B2) natural language level. While the effect on the resulting code proficiency was model-dependent, we found that higher-proficiency prompts consistently yielded more correct code across all models. These results demonstrate that natural language proficiency is a key lever for controlling code generation, helping developers tailor AI output and improve the reliability of solutions. |
| title | How Natural Language Proficiency Shapes GenAI Code for Software Engineering Tasks |
| topic | Software Engineering Programming Languages |
| url | https://arxiv.org/abs/2511.04115 |