DOCE: Finding the Sweet Spot for Execution-Based Code Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912074228039680 |
|---|---|
| author | Li, Haau-Sing Fernandes, Patrick Gurevych, Iryna Martins, André F. T. |
| author_facet | Li, Haau-Sing Fernandes, Patrick Gurevych, Iryna Martins, André F. T. |
| contents | Recently, a diverse set of decoding and reranking procedures have been shown effective for LLM-based code generation. However, a comprehensive framework that links and experimentally compares these methods is missing. We address this by proposing Decoding Objectives for Code Execution, a comprehensive framework that includes candidate generation, $n$-best reranking, minimum Bayes risk (MBR) decoding, and self-debugging as the core components. We then study the contributions of these components through execution-based evaluation metrics. Our findings highlight the importance of execution-based methods and the difference gap between execution-based and execution-free methods. Furthermore, we assess the impact of filtering based on trial unit tests, a simple and effective strategy that has been often overlooked in prior works. We also propose self-debugging on multiple candidates, obtaining state-of-the-art performance on reranking for code generation. We expect our framework to provide a solid guideline for future research on code generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_13745 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DOCE: Finding the Sweet Spot for Execution-Based Code Generation Li, Haau-Sing Fernandes, Patrick Gurevych, Iryna Martins, André F. T. Computation and Language Artificial Intelligence Programming Languages Recently, a diverse set of decoding and reranking procedures have been shown effective for LLM-based code generation. However, a comprehensive framework that links and experimentally compares these methods is missing. We address this by proposing Decoding Objectives for Code Execution, a comprehensive framework that includes candidate generation, $n$-best reranking, minimum Bayes risk (MBR) decoding, and self-debugging as the core components. We then study the contributions of these components through execution-based evaluation metrics. Our findings highlight the importance of execution-based methods and the difference gap between execution-based and execution-free methods. Furthermore, we assess the impact of filtering based on trial unit tests, a simple and effective strategy that has been often overlooked in prior works. We also propose self-debugging on multiple candidates, obtaining state-of-the-art performance on reranking for code generation. We expect our framework to provide a solid guideline for future research on code generation. |
| title | DOCE: Finding the Sweet Spot for Execution-Based Code Generation |
| topic | Computation and Language Artificial Intelligence Programming Languages |
| url | https://arxiv.org/abs/2408.13745 |