LLaDA2.0: Scaling Up Diffusion Language Models to 100B
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914218357293056 |
|---|---|
| author | Bie, Tiwei Cao, Maosong Chen, Kun Du, Lun Gong, Mingliang Gong, Zhuochen Gu, Yanmei Hu, Jiaqi Huang, Zenan Lan, Zhenzhong Li, Chengxi Li, Chongxuan Li, Jianguo Li, Zehuan Liu, Huabin Liu, Lin Lu, Guoshan Lu, Xiaocheng Ma, Yuxin Tan, Jianfeng Wei, Lanning Wen, Ji-Rong Xing, Yipeng Zhang, Xiaolu Zhao, Junbo Zheng, Da Zhou, Jun Zhou, Junlin Zhou, Zhanchao Zhu, Liwang Zhuang, Yihong |
| author_facet | Bie, Tiwei Cao, Maosong Chen, Kun Du, Lun Gong, Mingliang Gong, Zhuochen Gu, Yanmei Hu, Jiaqi Huang, Zenan Lan, Zhenzhong Li, Chengxi Li, Chongxuan Li, Jianguo Li, Zehuan Liu, Huabin Liu, Lin Lu, Guoshan Lu, Xiaocheng Ma, Yuxin Tan, Jianfeng Wei, Lanning Wen, Ji-Rong Xing, Yipeng Zhang, Xiaolu Zhao, Junbo Zheng, Da Zhou, Jun Zhou, Junlin Zhou, Zhanchao Zhu, Liwang Zhuang, Yihong |
| contents | This paper presents LLaDA2.0 -- a tuple of discrete diffusion large language models (dLLM) scaling up to 100B total parameters through systematic conversion from auto-regressive (AR) models -- establishing a new paradigm for frontier-scale deployment. Instead of costly training from scratch, LLaDA2.0 upholds knowledge inheritance, progressive adaption and efficiency-aware design principle, and seamless converts a pre-trained AR model into dLLM with a novel 3-phase block-level WSD based training scheme: progressive increasing block-size in block diffusion (warm-up), large-scale full-sequence diffusion (stable) and reverting back to compact-size block diffusion (decay). Along with post-training alignment with SFT and DPO, we obtain LLaDA2.0-mini (16B) and LLaDA2.0-flash (100B), two instruction-tuned Mixture-of-Experts (MoE) variants optimized for practical deployment. By preserving the advantages of parallel decoding, these models deliver superior performance and efficiency at the frontier scale. Both models were open-sourced. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_15745 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LLaDA2.0: Scaling Up Diffusion Language Models to 100B Bie, Tiwei Cao, Maosong Chen, Kun Du, Lun Gong, Mingliang Gong, Zhuochen Gu, Yanmei Hu, Jiaqi Huang, Zenan Lan, Zhenzhong Li, Chengxi Li, Chongxuan Li, Jianguo Li, Zehuan Liu, Huabin Liu, Lin Lu, Guoshan Lu, Xiaocheng Ma, Yuxin Tan, Jianfeng Wei, Lanning Wen, Ji-Rong Xing, Yipeng Zhang, Xiaolu Zhao, Junbo Zheng, Da Zhou, Jun Zhou, Junlin Zhou, Zhanchao Zhu, Liwang Zhuang, Yihong Machine Learning Artificial Intelligence Computation and Language This paper presents LLaDA2.0 -- a tuple of discrete diffusion large language models (dLLM) scaling up to 100B total parameters through systematic conversion from auto-regressive (AR) models -- establishing a new paradigm for frontier-scale deployment. Instead of costly training from scratch, LLaDA2.0 upholds knowledge inheritance, progressive adaption and efficiency-aware design principle, and seamless converts a pre-trained AR model into dLLM with a novel 3-phase block-level WSD based training scheme: progressive increasing block-size in block diffusion (warm-up), large-scale full-sequence diffusion (stable) and reverting back to compact-size block diffusion (decay). Along with post-training alignment with SFT and DPO, we obtain LLaDA2.0-mini (16B) and LLaDA2.0-flash (100B), two instruction-tuned Mixture-of-Experts (MoE) variants optimized for practical deployment. By preserving the advantages of parallel decoding, these models deliver superior performance and efficiency at the frontier scale. Both models were open-sourced. |
| title | LLaDA2.0: Scaling Up Diffusion Language Models to 100B |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2512.15745 |