Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.11103 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914409148841984 |
|---|---|
| author | Zeng, Zhiyuan Zhang, Yichi Shan, Yong Hua, Kai Fang, Siyuan Liu, Zhaiyu Liu, Jiaheng Wang, Haozhe Zheng, Yining Ding, Ming Shen, Ke Zhang, Ge Huang, Wenhao Qiu, Xipeng |
| author_facet | Zeng, Zhiyuan Zhang, Yichi Shan, Yong Hua, Kai Fang, Siyuan Liu, Zhaiyu Liu, Jiaheng Wang, Haozhe Zheng, Yining Ding, Ming Shen, Ke Zhang, Ge Huang, Wenhao Qiu, Xipeng |
| contents | While Large Language Models (LLMs) have achieved remarkable success in code generation, they often struggle with the deep, long-horizon reasoning required for complex software engineering. We attribute this limitation to the nature of standard pre-training data: static software repositories represent only the terminal state of an intricate intellectual process, abstracting away the intermediate planning, debugging, and iterative refinement. To bridge this gap, we propose a novel paradigm: understanding via reconstruction. We hypothesize that reverse-engineering the latent agentic trajectories -- the planning, reasoning, and debugging steps -- behind static repositories provides a far richer supervision signal than raw code alone. To operationalize this, we introduce a framework that synthesizes these trajectories using a multi-agent simulation. This process is grounded in the structural realities of the source repositories (e.g., dependency graphs and file hierarchies) to ensure fidelity. Furthermore, to guarantee the logical rigor of the synthetic data, we employ a search-based optimization technique that iteratively refines the Chain-of-Thought (CoT) reasoning to maximize the likelihood of the ground-truth code. Empirical results demonstrate that continuous pre-training on these reconstructed trajectories significantly enhances Llama-3-8B's performance across diverse benchmarks, including long-context understanding, coding proficiency, and agentic capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_11103 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Understanding by Reconstruction: Reversing the Software Development Process for LLM Pretraining Zeng, Zhiyuan Zhang, Yichi Shan, Yong Hua, Kai Fang, Siyuan Liu, Zhaiyu Liu, Jiaheng Wang, Haozhe Zheng, Yining Ding, Ming Shen, Ke Zhang, Ge Huang, Wenhao Qiu, Xipeng Software Engineering While Large Language Models (LLMs) have achieved remarkable success in code generation, they often struggle with the deep, long-horizon reasoning required for complex software engineering. We attribute this limitation to the nature of standard pre-training data: static software repositories represent only the terminal state of an intricate intellectual process, abstracting away the intermediate planning, debugging, and iterative refinement. To bridge this gap, we propose a novel paradigm: understanding via reconstruction. We hypothesize that reverse-engineering the latent agentic trajectories -- the planning, reasoning, and debugging steps -- behind static repositories provides a far richer supervision signal than raw code alone. To operationalize this, we introduce a framework that synthesizes these trajectories using a multi-agent simulation. This process is grounded in the structural realities of the source repositories (e.g., dependency graphs and file hierarchies) to ensure fidelity. Furthermore, to guarantee the logical rigor of the synthetic data, we employ a search-based optimization technique that iteratively refines the Chain-of-Thought (CoT) reasoning to maximize the likelihood of the ground-truth code. Empirical results demonstrate that continuous pre-training on these reconstructed trajectories significantly enhances Llama-3-8B's performance across diverse benchmarks, including long-context understanding, coding proficiency, and agentic capabilities. |
| title | Understanding by Reconstruction: Reversing the Software Development Process for LLM Pretraining |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2603.11103 |