What I cannot execute, I do not understand: Training and Evaluating LLMs on Program Execution Traces
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909531256127488 |
|---|---|
| author | Armengol-Estapé, Jordi Carbonneaux, Quentin Zhang, Tianjun Markosyan, Aram H. Seeker, Volker Cummins, Chris Kambadur, Melanie O'Boyle, Michael F. P. Wang, Sida Synnaeve, Gabriel Leather, Hugh James |
| author_facet | Armengol-Estapé, Jordi Carbonneaux, Quentin Zhang, Tianjun Markosyan, Aram H. Seeker, Volker Cummins, Chris Kambadur, Melanie O'Boyle, Michael F. P. Wang, Sida Synnaeve, Gabriel Leather, Hugh James |
| contents | Code generation and understanding are critical capabilities for large language models (LLMs). Thus, most LLMs are pretrained and fine-tuned on code data. However, these datasets typically treat code as static strings and rarely exploit the dynamic information about their execution. Building upon previous work on trace modeling, we study Execution Tuning (E.T.), a training procedure in which we explicitly model real-world program execution traces without requiring manual test annotations. We train and evaluate models on different execution trace granularities (line and instruction-level) and strategies on the task of output prediction, obtaining around 80% accuracy on CruxEval and MBPP, and showing the advantages of dynamic scratchpads (i.e., self-contained intermediate computations updated by the model rather than accumulated as a history of past computations) on long executions (up to 14k steps). Finally, we discuss E.T.'s practical applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_05703 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | What I cannot execute, I do not understand: Training and Evaluating LLMs on Program Execution Traces Armengol-Estapé, Jordi Carbonneaux, Quentin Zhang, Tianjun Markosyan, Aram H. Seeker, Volker Cummins, Chris Kambadur, Melanie O'Boyle, Michael F. P. Wang, Sida Synnaeve, Gabriel Leather, Hugh James Machine Learning Artificial Intelligence Programming Languages Code generation and understanding are critical capabilities for large language models (LLMs). Thus, most LLMs are pretrained and fine-tuned on code data. However, these datasets typically treat code as static strings and rarely exploit the dynamic information about their execution. Building upon previous work on trace modeling, we study Execution Tuning (E.T.), a training procedure in which we explicitly model real-world program execution traces without requiring manual test annotations. We train and evaluate models on different execution trace granularities (line and instruction-level) and strategies on the task of output prediction, obtaining around 80% accuracy on CruxEval and MBPP, and showing the advantages of dynamic scratchpads (i.e., self-contained intermediate computations updated by the model rather than accumulated as a history of past computations) on long executions (up to 14k steps). Finally, we discuss E.T.'s practical applications. |
| title | What I cannot execute, I do not understand: Training and Evaluating LLMs on Program Execution Traces |
| topic | Machine Learning Artificial Intelligence Programming Languages |
| url | https://arxiv.org/abs/2503.05703 |