Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.07464 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911258494631936 |
|---|---|
| author | Lim, Junghwan Lee, Sungmin Kim, Dongseok Kim, Taehyun Park, Eunhwan Lee, Jeesoo Lee, Jeongdoo Lee, Junhyeok Cheung, Wai Ting Choi, Dahye Her, Jaeheui Huh, Jaeyeon Jung, Hanbin Kang, Changjin Kim, Beomgyu Kim, Minjae Kim, Taewhan Kim, Youngrok Kweon, Hyukjin Lee, Haesol Lee, Kungyu Oh, Dongpin Park, Yeongjae Ryu, Bokki Weon, Dongjoo |
| author_facet | Lim, Junghwan Lee, Sungmin Kim, Dongseok Kim, Taehyun Park, Eunhwan Lee, Jeesoo Lee, Jeongdoo Lee, Junhyeok Cheung, Wai Ting Choi, Dahye Her, Jaeheui Huh, Jaeyeon Jung, Hanbin Kang, Changjin Kim, Beomgyu Kim, Minjae Kim, Taewhan Kim, Youngrok Kweon, Hyukjin Lee, Haesol Lee, Kungyu Oh, Dongpin Park, Yeongjae Ryu, Bokki Weon, Dongjoo |
| contents | We introduce Motif-2-12.7B, a new open-weight foundation model that pushes the efficiency frontier of large language models by combining architectural innovation with system-level optimization. Designed for scalable language understanding and robust instruction generalization under constrained compute budgets, Motif-2-12.7B builds upon Motif-2.6B with the integration of Grouped Differential Attention (GDA), which improves representational efficiency by disentangling signal and noise-control attention pathways. The model is pre-trained on 5.5 trillion tokens spanning diverse linguistic, mathematical, scientific, and programming domains using a curriculum-driven data scheduler that gradually changes the data composition ratio. The training system leverages the MuonClip optimizer alongside custom high-performance kernels, including fused PolyNorm activations and the Parallel Muon algorithm, yielding significant throughput and memory efficiency gains in large-scale distributed environments. Post-training employs a three-stage supervised fine-tuning pipeline that successively enhances general instruction adherence, compositional understanding, and linguistic precision. Motif-2-12.7B demonstrates competitive performance across diverse benchmarks, showing that thoughtful architectural scaling and optimized training design can rival the capabilities of much larger models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_07464 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Motif 2 12.7B technical report Lim, Junghwan Lee, Sungmin Kim, Dongseok Kim, Taehyun Park, Eunhwan Lee, Jeesoo Lee, Jeongdoo Lee, Junhyeok Cheung, Wai Ting Choi, Dahye Her, Jaeheui Huh, Jaeyeon Jung, Hanbin Kang, Changjin Kim, Beomgyu Kim, Minjae Kim, Taewhan Kim, Youngrok Kweon, Hyukjin Lee, Haesol Lee, Kungyu Oh, Dongpin Park, Yeongjae Ryu, Bokki Weon, Dongjoo Computation and Language Artificial Intelligence We introduce Motif-2-12.7B, a new open-weight foundation model that pushes the efficiency frontier of large language models by combining architectural innovation with system-level optimization. Designed for scalable language understanding and robust instruction generalization under constrained compute budgets, Motif-2-12.7B builds upon Motif-2.6B with the integration of Grouped Differential Attention (GDA), which improves representational efficiency by disentangling signal and noise-control attention pathways. The model is pre-trained on 5.5 trillion tokens spanning diverse linguistic, mathematical, scientific, and programming domains using a curriculum-driven data scheduler that gradually changes the data composition ratio. The training system leverages the MuonClip optimizer alongside custom high-performance kernels, including fused PolyNorm activations and the Parallel Muon algorithm, yielding significant throughput and memory efficiency gains in large-scale distributed environments. Post-training employs a three-stage supervised fine-tuning pipeline that successively enhances general instruction adherence, compositional understanding, and linguistic precision. Motif-2-12.7B demonstrates competitive performance across diverse benchmarks, showing that thoughtful architectural scaling and optimized training design can rival the capabilities of much larger models. |
| title | Motif 2 12.7B technical report |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2511.07464 |