From Effectiveness to Efficiency: Uncovering Linguistic Bias in Large Language Model-based Code Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910923497668608 |
|---|---|
| author | Jiang, Weipeng Gao, Xuanqi Zhai, Juan Ma, Shiqing Zhang, Xiaoyu Lei, Ziyan Shen, Chao |
| author_facet | Jiang, Weipeng Gao, Xuanqi Zhai, Juan Ma, Shiqing Zhang, Xiaoyu Lei, Ziyan Shen, Chao |
| contents | Large Language Models (LLMs) have demonstrated promising capabilities for code generation. While existing benchmarks evaluate the correctness and efficiency of LLM-generated code, the potential linguistic bias - where code quality varies based on the natural language used to describe programming tasks - remains underexplored. In this paper, we aim to investigate this linguistic bias through the lens of English and Chinese. To facilitate our investigation, we present a unified evaluation framework comprising a curated dataset of 52 Python programming questions with parallel bilingual task descriptions, automated correctness verification, and efficiency quantification tools based on runtime complexity estimation. Based on this framework, we conduct the first empirical study towards the linguistic bias in LLM-generated code on eight popular LCGMs, as well as GPT-3.5-Turbo and GPT-4. We observe that these LCGM-generated code show different correctness on an average of 12% bilingual programming tasks, where 39% also exhibits diverse efficiency. Our findings indicate that LLMs commonly exhibit linguistic bias for code generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_00602 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | From Effectiveness to Efficiency: Uncovering Linguistic Bias in Large Language Model-based Code Generation Jiang, Weipeng Gao, Xuanqi Zhai, Juan Ma, Shiqing Zhang, Xiaoyu Lei, Ziyan Shen, Chao Software Engineering Programming Languages Large Language Models (LLMs) have demonstrated promising capabilities for code generation. While existing benchmarks evaluate the correctness and efficiency of LLM-generated code, the potential linguistic bias - where code quality varies based on the natural language used to describe programming tasks - remains underexplored. In this paper, we aim to investigate this linguistic bias through the lens of English and Chinese. To facilitate our investigation, we present a unified evaluation framework comprising a curated dataset of 52 Python programming questions with parallel bilingual task descriptions, automated correctness verification, and efficiency quantification tools based on runtime complexity estimation. Based on this framework, we conduct the first empirical study towards the linguistic bias in LLM-generated code on eight popular LCGMs, as well as GPT-3.5-Turbo and GPT-4. We observe that these LCGM-generated code show different correctness on an average of 12% bilingual programming tasks, where 39% also exhibits diverse efficiency. Our findings indicate that LLMs commonly exhibit linguistic bias for code generation. |
| title | From Effectiveness to Efficiency: Uncovering Linguistic Bias in Large Language Model-based Code Generation |
| topic | Software Engineering Programming Languages |
| url | https://arxiv.org/abs/2406.00602 |