Aligning CodeLLMs with Direct Preference Optimization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909362172198912 |
|---|---|
| author | Miao, Yibo Gao, Bofei Quan, Shanghaoran Lin, Junyang Zan, Daoguang Liu, Jiaheng Yang, Jian Liu, Tianyu Deng, Zhijie |
| author_facet | Miao, Yibo Gao, Bofei Quan, Shanghaoran Lin, Junyang Zan, Daoguang Liu, Jiaheng Yang, Jian Liu, Tianyu Deng, Zhijie |
| contents | The last year has witnessed the rapid progress of large language models (LLMs) across diverse domains. Among them, CodeLLMs have garnered particular attention because they can not only assist in completing various programming tasks but also represent the decision-making and logical reasoning capabilities of LLMs. However, current CodeLLMs mainly focus on pre-training and supervised fine-tuning scenarios, leaving the alignment stage, which is important for post-training LLMs, under-explored. This work first identifies that the commonly used PPO algorithm may be suboptimal for the alignment of CodeLLM because the involved reward rules are routinely coarse-grained and potentially flawed. We then advocate addressing this using the DPO algorithm. Based on only preference data pairs, DPO can render the model rank data automatically, giving rise to a fine-grained rewarding pattern more robust than human intervention. We also contribute a pipeline for collecting preference pairs for DPO on CodeLLMs. Studies show that our method significantly improves the performance of existing CodeLLMs on benchmarks such as MBPP and HumanEval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_18585 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Aligning CodeLLMs with Direct Preference Optimization Miao, Yibo Gao, Bofei Quan, Shanghaoran Lin, Junyang Zan, Daoguang Liu, Jiaheng Yang, Jian Liu, Tianyu Deng, Zhijie Artificial Intelligence Machine Learning The last year has witnessed the rapid progress of large language models (LLMs) across diverse domains. Among them, CodeLLMs have garnered particular attention because they can not only assist in completing various programming tasks but also represent the decision-making and logical reasoning capabilities of LLMs. However, current CodeLLMs mainly focus on pre-training and supervised fine-tuning scenarios, leaving the alignment stage, which is important for post-training LLMs, under-explored. This work first identifies that the commonly used PPO algorithm may be suboptimal for the alignment of CodeLLM because the involved reward rules are routinely coarse-grained and potentially flawed. We then advocate addressing this using the DPO algorithm. Based on only preference data pairs, DPO can render the model rank data automatically, giving rise to a fine-grained rewarding pattern more robust than human intervention. We also contribute a pipeline for collecting preference pairs for DPO on CodeLLMs. Studies show that our method significantly improves the performance of existing CodeLLMs on benchmarks such as MBPP and HumanEval. |
| title | Aligning CodeLLMs with Direct Preference Optimization |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2410.18585 |