Code Needs Comments: Enhancing Code LLMs with Comment Augmentation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917594382991360 |
|---|---|
| author | Song, Demin Guo, Honglin Zhou, Yunhua Xing, Shuhao Wang, Yudong Song, Zifan Zhang, Wenwei Guo, Qipeng Yan, Hang Qiu, Xipeng Lin, Dahua |
| author_facet | Song, Demin Guo, Honglin Zhou, Yunhua Xing, Shuhao Wang, Yudong Song, Zifan Zhang, Wenwei Guo, Qipeng Yan, Hang Qiu, Xipeng Lin, Dahua |
| contents | The programming skill is one crucial ability for Large Language Models (LLMs), necessitating a deep understanding of programming languages (PLs) and their correlation with natural languages (NLs). We examine the impact of pre-training data on code-focused LLMs' performance by assessing the comment density as a measure of PL-NL alignment. Given the scarcity of code-comment aligned data in pre-training corpora, we introduce a novel data augmentation method that generates comments for existing code, coupled with a data filtering strategy that filters out code data poorly correlated with natural language. We conducted experiments on three code-focused LLMs and observed consistent improvements in performance on two widely-used programming skill benchmarks. Notably, the model trained on the augmented data outperformed both the model used for generating comments and the model further trained on the data without augmentation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_13013 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Code Needs Comments: Enhancing Code LLMs with Comment Augmentation Song, Demin Guo, Honglin Zhou, Yunhua Xing, Shuhao Wang, Yudong Song, Zifan Zhang, Wenwei Guo, Qipeng Yan, Hang Qiu, Xipeng Lin, Dahua Computation and Language The programming skill is one crucial ability for Large Language Models (LLMs), necessitating a deep understanding of programming languages (PLs) and their correlation with natural languages (NLs). We examine the impact of pre-training data on code-focused LLMs' performance by assessing the comment density as a measure of PL-NL alignment. Given the scarcity of code-comment aligned data in pre-training corpora, we introduce a novel data augmentation method that generates comments for existing code, coupled with a data filtering strategy that filters out code data poorly correlated with natural language. We conducted experiments on three code-focused LLMs and observed consistent improvements in performance on two widely-used programming skill benchmarks. Notably, the model trained on the augmented data outperformed both the model used for generating comments and the model further trained on the data without augmentation. |
| title | Code Needs Comments: Enhancing Code LLMs with Comment Augmentation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.13013 |