CoDA: Coding LM via Diffusion Adaptation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908574853103616 |
|---|---|
| author | Chen, Haolin Wang, Shiyu Qin, Can Pang, Bo Liu, Zuxin Qiu, Jielin Zhang, Jianguo Zhou, Yingbo Chen, Zeyuan Xu, Ran Heinecke, Shelby Savarese, Silvio Xiong, Caiming Wang, Huan Yao, Weiran |
| author_facet | Chen, Haolin Wang, Shiyu Qin, Can Pang, Bo Liu, Zuxin Qiu, Jielin Zhang, Jianguo Zhou, Yingbo Chen, Zeyuan Xu, Ran Heinecke, Shelby Savarese, Silvio Xiong, Caiming Wang, Huan Yao, Weiran |
| contents | Diffusion language models promise bidirectional context and infilling capabilities that autoregressive coders lack, yet practical systems remain heavyweight. We introduce CoDA, a 1.7B-parameter diffusion coder trained on TPU with a fully open-source training pipeline. CoDA pairs large-scale diffusion pre-training with code-centric mid-training and instruction tuning, enabling confidence-guided sampling that keeps inference latency competitive. On Humaneval, MBPP, and EvalPlus, CoDA-1.7B-Instruct matches or surpasses diffusion models up to 7B parameters. Our release includes model checkpoints, evaluation harnesses, and TPU training pipelines to accelerate research on lightweight diffusion-based coding assistants. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_03270 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CoDA: Coding LM via Diffusion Adaptation Chen, Haolin Wang, Shiyu Qin, Can Pang, Bo Liu, Zuxin Qiu, Jielin Zhang, Jianguo Zhou, Yingbo Chen, Zeyuan Xu, Ran Heinecke, Shelby Savarese, Silvio Xiong, Caiming Wang, Huan Yao, Weiran Machine Learning Artificial Intelligence I.2.7 Diffusion language models promise bidirectional context and infilling capabilities that autoregressive coders lack, yet practical systems remain heavyweight. We introduce CoDA, a 1.7B-parameter diffusion coder trained on TPU with a fully open-source training pipeline. CoDA pairs large-scale diffusion pre-training with code-centric mid-training and instruction tuning, enabling confidence-guided sampling that keeps inference latency competitive. On Humaneval, MBPP, and EvalPlus, CoDA-1.7B-Instruct matches or surpasses diffusion models up to 7B parameters. Our release includes model checkpoints, evaluation harnesses, and TPU training pipelines to accelerate research on lightweight diffusion-based coding assistants. |
| title | CoDA: Coding LM via Diffusion Adaptation |
| topic | Machine Learning Artificial Intelligence I.2.7 |
| url | https://arxiv.org/abs/2510.03270 |