Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.07612 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912754146738176 |
|---|---|
| author | Luo, Kairong Sun, Zhenbo Shi, Xinyu Chen, Shengqi Yu, Bowen Chen, Yunyi Dang, Chenyi Tao, Hengtao Wang, Hui Liu, Fangming Lyu, Kaifeng Chen, Wenguang |
| author_facet | Luo, Kairong Sun, Zhenbo Shi, Xinyu Chen, Shengqi Yu, Bowen Chen, Yunyi Dang, Chenyi Tao, Hengtao Wang, Hui Liu, Fangming Lyu, Kaifeng Chen, Wenguang |
| contents | The rapid advancement of Large Language Models (LLMs) has resulted in a significant knowledge gap between the open-source community and industry, primarily because the latter relies on closed-source, high-quality data and training recipes. To address this, we introduce PCMind-2.1-Kaiyuan-2B, a fully open-source 2-billion-parameter model focused on improving training efficiency and effectiveness under resource constraints. Our methodology includes three key innovations: a Quantile Data Benchmarking method for systematically comparing heterogeneous open-source datasets and providing insights on data mixing strategies; a Strategic Selective Repetition scheme within a multi-phase paradigm to effectively leverage sparse, high-quality data; and a Multi-Domain Curriculum Training policy that orders samples by quality. Supported by a highly optimized data preprocessing pipeline and architectural modifications for FP16 stability, Kaiyuan-2B achieves performance competitive with state-of-the-art fully open-source models, demonstrating practical and scalable solutions for resource-limited pretraining. We release all assets (including model weights, data, and code) under Apache 2.0 license at https://huggingface.co/thu-pacman/PCMind-2.1-Kaiyuan-2B. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_07612 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PCMind-2.1-Kaiyuan-2B Technical Report Luo, Kairong Sun, Zhenbo Shi, Xinyu Chen, Shengqi Yu, Bowen Chen, Yunyi Dang, Chenyi Tao, Hengtao Wang, Hui Liu, Fangming Lyu, Kaifeng Chen, Wenguang Computation and Language Artificial Intelligence Machine Learning The rapid advancement of Large Language Models (LLMs) has resulted in a significant knowledge gap between the open-source community and industry, primarily because the latter relies on closed-source, high-quality data and training recipes. To address this, we introduce PCMind-2.1-Kaiyuan-2B, a fully open-source 2-billion-parameter model focused on improving training efficiency and effectiveness under resource constraints. Our methodology includes three key innovations: a Quantile Data Benchmarking method for systematically comparing heterogeneous open-source datasets and providing insights on data mixing strategies; a Strategic Selective Repetition scheme within a multi-phase paradigm to effectively leverage sparse, high-quality data; and a Multi-Domain Curriculum Training policy that orders samples by quality. Supported by a highly optimized data preprocessing pipeline and architectural modifications for FP16 stability, Kaiyuan-2B achieves performance competitive with state-of-the-art fully open-source models, demonstrating practical and scalable solutions for resource-limited pretraining. We release all assets (including model weights, data, and code) under Apache 2.0 license at https://huggingface.co/thu-pacman/PCMind-2.1-Kaiyuan-2B. |
| title | PCMind-2.1-Kaiyuan-2B Technical Report |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2512.07612 |