A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910959606431744 |
|---|---|
| author | Katzy, Jonathan Huang, Yongcheng Panchu, Gopal-Raj Ziemlewski, Maksym Loizides, Paris Vermeulen, Sander van Deursen, Arie Izadi, Maliheh |
| author_facet | Katzy, Jonathan Huang, Yongcheng Panchu, Gopal-Raj Ziemlewski, Maksym Loizides, Paris Vermeulen, Sander van Deursen, Arie Izadi, Maliheh |
| contents | Large Language Models are essential coding assistants, yet their training is predominantly English-centric. In this study, we evaluate the performance of code language models in non-English contexts, identifying challenges in their adoption and integration into multilingual workflows. We conduct an open-coding study to analyze errors in code comments generated by five state-of-the-art code models, CodeGemma, CodeLlama, CodeQwen1.5, GraniteCode, and StarCoder2 across five natural languages: Chinese, Dutch, English, Greek, and Polish. Our study yields a dataset of 12,500 labeled generations, which we publicly release. We then assess the reliability of standard metrics in capturing comment \textit{correctness} across languages and evaluate their trustworthiness as judgment criteria. Through our open-coding investigation, we identified a taxonomy of 26 distinct error categories in model-generated code comments. They highlight variations in language cohesion, informativeness, and syntax adherence across different natural languages. Our analysis shows that, while these models frequently produce partially correct comments, modern neural metrics fail to reliably differentiate meaningful completions from random noise. Notably, the significant score overlap between expert-rated correct and incorrect comments calls into question the effectiveness of these metrics in assessing generated comments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_15469 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics Katzy, Jonathan Huang, Yongcheng Panchu, Gopal-Raj Ziemlewski, Maksym Loizides, Paris Vermeulen, Sander van Deursen, Arie Izadi, Maliheh Software Engineering Artificial Intelligence Large Language Models are essential coding assistants, yet their training is predominantly English-centric. In this study, we evaluate the performance of code language models in non-English contexts, identifying challenges in their adoption and integration into multilingual workflows. We conduct an open-coding study to analyze errors in code comments generated by five state-of-the-art code models, CodeGemma, CodeLlama, CodeQwen1.5, GraniteCode, and StarCoder2 across five natural languages: Chinese, Dutch, English, Greek, and Polish. Our study yields a dataset of 12,500 labeled generations, which we publicly release. We then assess the reliability of standard metrics in capturing comment \textit{correctness} across languages and evaluate their trustworthiness as judgment criteria. Through our open-coding investigation, we identified a taxonomy of 26 distinct error categories in model-generated code comments. They highlight variations in language cohesion, informativeness, and syntax adherence across different natural languages. Our analysis shows that, while these models frequently produce partially correct comments, modern neural metrics fail to reliably differentiate meaningful completions from random noise. Notably, the significant score overlap between expert-rated correct and incorrect comments calls into question the effectiveness of these metrics in assessing generated comments. |
| title | A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2505.15469 |