Does Few-Shot Learning Help LLM Performance in Code Synthesis?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909415084392448 |
|---|---|
| author | Xu, Derek Xie, Tong Xia, Botao Li, Haoyu Bai, Yunsheng Sun, Yizhou Wang, Wei |
| author_facet | Xu, Derek Xie, Tong Xia, Botao Li, Haoyu Bai, Yunsheng Sun, Yizhou Wang, Wei |
| contents | Large language models (LLMs) have made significant strides at code generation through improved model design, training, and chain-of-thought. However, prompt-level optimizations remain an important yet under-explored aspect of LLMs for coding. This work focuses on the few-shot examples present in most code generation prompts, offering a systematic study on whether few-shot examples improve LLM's coding capabilities, which few-shot examples have the largest impact, and how to select impactful examples. Our work offers 2 approaches for selecting few-shot examples, a model-free method, CODEEXEMPLAR-FREE, and a model-based method, CODEEXEMPLAR-BASED. The 2 methods offer a trade-off between improved performance and reliance on training data and interpretability. Both methods significantly improve CodeLlama's coding ability across the popular HumanEval+ coding benchmark. In summary, our work provides valuable insights into how to pick few-shot examples in code generation prompts to improve LLM code generation capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_02906 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Does Few-Shot Learning Help LLM Performance in Code Synthesis? Xu, Derek Xie, Tong Xia, Botao Li, Haoyu Bai, Yunsheng Sun, Yizhou Wang, Wei Software Engineering Artificial Intelligence Computation and Language Machine Learning Large language models (LLMs) have made significant strides at code generation through improved model design, training, and chain-of-thought. However, prompt-level optimizations remain an important yet under-explored aspect of LLMs for coding. This work focuses on the few-shot examples present in most code generation prompts, offering a systematic study on whether few-shot examples improve LLM's coding capabilities, which few-shot examples have the largest impact, and how to select impactful examples. Our work offers 2 approaches for selecting few-shot examples, a model-free method, CODEEXEMPLAR-FREE, and a model-based method, CODEEXEMPLAR-BASED. The 2 methods offer a trade-off between improved performance and reliance on training data and interpretability. Both methods significantly improve CodeLlama's coding ability across the popular HumanEval+ coding benchmark. In summary, our work provides valuable insights into how to pick few-shot examples in code generation prompts to improve LLM code generation capabilities. |
| title | Does Few-Shot Learning Help LLM Performance in Code Synthesis? |
| topic | Software Engineering Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2412.02906 |