NExT: Teaching Large Language Models to Reason about Code Execution
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910418584207360 |
|---|---|
| author | Ni, Ansong Allamanis, Miltiadis Cohan, Arman Deng, Yinlin Shi, Kensen Sutton, Charles Yin, Pengcheng |
| author_facet | Ni, Ansong Allamanis, Miltiadis Cohan, Arman Deng, Yinlin Shi, Kensen Sutton, Charles Yin, Pengcheng |
| contents | A fundamental skill among human developers is the ability to understand and reason about program execution. As an example, a programmer can mentally simulate code execution in natural language to debug and repair code (aka. rubber duck debugging). However, large language models (LLMs) of code are typically trained on the surface textual form of programs, thus may lack a semantic understanding of how programs execute at run-time. To address this issue, we propose NExT, a method to teach LLMs to inspect the execution traces of programs (variable states of executed lines) and reason about their run-time behavior through chain-of-thought (CoT) rationales. Specifically, NExT uses self-training to bootstrap a synthetic training set of execution-aware rationales that lead to correct task solutions (e.g., fixed programs) without laborious manual annotation. Experiments on program repair tasks based on MBPP and HumanEval demonstrate that NExT improves the fix rate of a PaLM 2 model, by 26.1% and 14.3% absolute, respectively, with significantly improved rationale quality as verified by automated metrics and human raters. Our model can also generalize to scenarios where program traces are absent at test-time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_14662 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | NExT: Teaching Large Language Models to Reason about Code Execution Ni, Ansong Allamanis, Miltiadis Cohan, Arman Deng, Yinlin Shi, Kensen Sutton, Charles Yin, Pengcheng Machine Learning Computation and Language Programming Languages Software Engineering A fundamental skill among human developers is the ability to understand and reason about program execution. As an example, a programmer can mentally simulate code execution in natural language to debug and repair code (aka. rubber duck debugging). However, large language models (LLMs) of code are typically trained on the surface textual form of programs, thus may lack a semantic understanding of how programs execute at run-time. To address this issue, we propose NExT, a method to teach LLMs to inspect the execution traces of programs (variable states of executed lines) and reason about their run-time behavior through chain-of-thought (CoT) rationales. Specifically, NExT uses self-training to bootstrap a synthetic training set of execution-aware rationales that lead to correct task solutions (e.g., fixed programs) without laborious manual annotation. Experiments on program repair tasks based on MBPP and HumanEval demonstrate that NExT improves the fix rate of a PaLM 2 model, by 26.1% and 14.3% absolute, respectively, with significantly improved rationale quality as verified by automated metrics and human raters. Our model can also generalize to scenarios where program traces are absent at test-time. |
| title | NExT: Teaching Large Language Models to Reason about Code Execution |
| topic | Machine Learning Computation and Language Programming Languages Software Engineering |
| url | https://arxiv.org/abs/2404.14662 |