Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2410.08105 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912313928318976 |
|---|---|
| author | Zheng, Kunhao Decugis, Juliette Gehring, Jonas Cohen, Taco Negrevergne, Benjamin Synnaeve, Gabriel |
| author_facet | Zheng, Kunhao Decugis, Juliette Gehring, Jonas Cohen, Taco Negrevergne, Benjamin Synnaeve, Gabriel |
| contents | Prompting techniques such as chain-of-thought have established themselves as a popular vehicle for improving the outputs of large language models (LLMs). For code generation, however, their exact mechanics and efficacy are under-explored. We thus investigate the effects of a wide range of prompting strategies with a focus on automatic re-prompting over multiple turns and computational requirements. After systematically decomposing reasoning, instruction, and execution feedback prompts, we conduct an extensive grid search on the competitive programming benchmarks CodeContests and TACO for multiple LLM families and sizes (Llama 3.0 and 3.1, 8B, 70B, 405B, and GPT-4o). Our study reveals strategies that consistently improve performance across all models with small and large sampling budgets. We then show how finetuning with such an optimal configuration allows models to internalize the induced reasoning process and obtain improvements in performance and scalability for multi-turn code generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_08105 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | What Makes Large Language Models Reason in (Multi-Turn) Code Generation? Zheng, Kunhao Decugis, Juliette Gehring, Jonas Cohen, Taco Negrevergne, Benjamin Synnaeve, Gabriel Computation and Language Prompting techniques such as chain-of-thought have established themselves as a popular vehicle for improving the outputs of large language models (LLMs). For code generation, however, their exact mechanics and efficacy are under-explored. We thus investigate the effects of a wide range of prompting strategies with a focus on automatic re-prompting over multiple turns and computational requirements. After systematically decomposing reasoning, instruction, and execution feedback prompts, we conduct an extensive grid search on the competitive programming benchmarks CodeContests and TACO for multiple LLM families and sizes (Llama 3.0 and 3.1, 8B, 70B, 405B, and GPT-4o). Our study reveals strategies that consistently improve performance across all models with small and large sampling budgets. We then show how finetuning with such an optimal configuration allows models to internalize the induced reasoning process and obtain improvements in performance and scalability for multi-turn code generation. |
| title | What Makes Large Language Models Reason in (Multi-Turn) Code Generation? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.08105 |