RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913694812733440 |
|---|---|
| author | Gehring, Jonas Zheng, Kunhao Copet, Jade Mella, Vegard Carbonneaux, Quentin Cohen, Taco Synnaeve, Gabriel |
| author_facet | Gehring, Jonas Zheng, Kunhao Copet, Jade Mella, Vegard Carbonneaux, Quentin Cohen, Taco Synnaeve, Gabriel |
| contents | Large language models (LLMs) deployed as agents solve user-specified tasks over multiple steps while keeping the required manual engagement to a minimum. Crucially, such LLMs need to ground their generations in any feedback obtained to reliably achieve the desired outcomes. We propose an end-to-end reinforcement learning method for teaching models to leverage execution feedback in the realm of code synthesis, where state-of-the-art LLMs struggle to improve code iteratively compared to independent sampling. We benchmark on competitive programming tasks, where we achieve new state-of-the art results with both small (8B parameters) and large (70B) models while reducing the amount of samples required by an order of magnitude. Our analysis of inference-time behavior demonstrates that our method produces LLMs that effectively leverage automatic feedback over multiple steps. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02089 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning Gehring, Jonas Zheng, Kunhao Copet, Jade Mella, Vegard Carbonneaux, Quentin Cohen, Taco Synnaeve, Gabriel Computation and Language Artificial Intelligence Large language models (LLMs) deployed as agents solve user-specified tasks over multiple steps while keeping the required manual engagement to a minimum. Crucially, such LLMs need to ground their generations in any feedback obtained to reliably achieve the desired outcomes. We propose an end-to-end reinforcement learning method for teaching models to leverage execution feedback in the realm of code synthesis, where state-of-the-art LLMs struggle to improve code iteratively compared to independent sampling. We benchmark on competitive programming tasks, where we achieve new state-of-the art results with both small (8B parameters) and large (70B) models while reducing the amount of samples required by an order of magnitude. Our analysis of inference-time behavior demonstrates that our method produces LLMs that effectively leverage automatic feedback over multiple steps. |
| title | RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.02089 |