Copertina

JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Cai, Aichen, Zhang, Anmeng, Li, Anyu, Zhang, Bo, Cai, Bohua, Li, Chang, Jiang, Changjian, Lu, Changkai, Xue, Chao, Liang, Chaocai, Zhang, Cheng, Liu, Dongkai, Wang, Fei, Huang, Guoqiang, Ke, Haijian, Lin, Han, Wang, Hao, Miao, Ji, Zhang, Jiacheng, Shi, Jialong, Zhu, Jifeng, Qian, Jingjing, Luo, Junhui, Xiong, Junwu, So, Lam, Huang, Liang, Ke, Ming, Li, Mingyang, Shi, Panfeng, Hao, Peng, Wang, Qi, Lai, Qian, Yuan, Qiaoqiao, Yin, Qingyu, Cao, Qiong, Wang, Qixiang, Bian, Rongcheng, Han, Rongduo, Zheng, Shaoqiang, Hu, Shi, Suo, Shi, Ren, Shijie, Zhang, Shijin, Fan, Shiying, Xie, Shuai, Zhang, Tianyi, Liu, Wei, Tan, Wentao, Meng, Xianghan, He, Xiaodong, Pan, Xing, Wang, Xiran, Peng, Xuyang, Zhang, Ya, Liu, Yang, Duan, Yangyang, Chen, Yanxu, Gong, Yicheng, Huang, Yidan, Liu, Yifei, Bai, Yinhao, Liu, Yongqiang, Zhang, Yuesong, Zhang, Yuqi, Xie, Zerui, Wang, Zhenfang, Shen, Zhennan, Liu, Zheyuan, Zeng, Zhuwei
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Computation and Language Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

_version_	1866915923332431872
author	Cai, Aichen Zhang, Anmeng Li, Anyu Zhang, Bo Cai, Bohua Li, Chang Jiang, Changjian Lu, Changkai Xue, Chao Liang, Chaocai Zhang, Cheng Liu, Dongkai Wang, Fei Huang, Guoqiang Ke, Haijian Lin, Han Wang, Hao Miao, Ji Zhang, Jiacheng Shi, Jialong Zhu, Jifeng Qian, Jingjing Luo, Junhui Xiong, Junwu So, Lam Huang, Liang Ke, Ming Li, Mingyang Shi, Panfeng Hao, Peng Wang, Qi Lai, Qian Yuan, Qiaoqiao Yin, Qingyu Cao, Qiong Wang, Qixiang Bian, Rongcheng Han, Rongduo Zheng, Shaoqiang Hu, Shi Suo, Shi Ren, Shijie Zhang, Shijin Fan, Shiying Xie, Shuai Zhang, Tianyi Liu, Wei Tan, Wentao Meng, Xianghan He, Xiaodong Pan, Xing Wang, Xiran Peng, Xuyang Zhang, Ya Liu, Yang Duan, Yangyang Chen, Yanxu Gong, Yicheng Huang, Yidan Liu, Yifei Bai, Yinhao Liu, Yongqiang Zhang, Yuesong Zhang, Yuqi Xie, Zerui Wang, Zhenfang Shen, Zhennan Liu, Zheyuan Zeng, Zhuwei
author_facet	Cai, Aichen Zhang, Anmeng Li, Anyu Zhang, Bo Cai, Bohua Li, Chang Jiang, Changjian Lu, Changkai Xue, Chao Liang, Chaocai Zhang, Cheng Liu, Dongkai Wang, Fei Huang, Guoqiang Ke, Haijian Lin, Han Wang, Hao Miao, Ji Zhang, Jiacheng Shi, Jialong Zhu, Jifeng Qian, Jingjing Luo, Junhui Xiong, Junwu So, Lam Huang, Liang Ke, Ming Li, Mingyang Shi, Panfeng Hao, Peng Wang, Qi Lai, Qian Yuan, Qiaoqiao Yin, Qingyu Cao, Qiong Wang, Qixiang Bian, Rongcheng Han, Rongduo Zheng, Shaoqiang Hu, Shi Suo, Shi Ren, Shijie Zhang, Shijin Fan, Shiying Xie, Shuai Zhang, Tianyi Liu, Wei Tan, Wentao Meng, Xianghan He, Xiaodong Pan, Xing Wang, Xiran Peng, Xuyang Zhang, Ya Liu, Yang Duan, Yangyang Chen, Yanxu Gong, Yicheng Huang, Yidan Liu, Yifei Bai, Yinhao Liu, Yongqiang Zhang, Yuesong Zhang, Yuqi Xie, Zerui Wang, Zhenfang Shen, Zhennan Liu, Zheyuan Zeng, Zhuwei
contents	We introduce JoyAI-LLM Flash, an efficient Mixture-of-Experts (MoE) language model designed to redefine the trade-off between strong performance and token efficiency in the sub-50B parameter regime. JoyAI-LLM Flash is pretrained on a massive corpus of 20 trillion tokens and further optimized through a rigorous post-training pipeline, including supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and large-scale reinforcement learning (RL) across diverse environments. To improve token efficiency, JoyAI-LLM Flash strategically balances \emph{thinking} and \emph{non-thinking} cognitive modes and introduces FiberPO, a novel RL algorithm inspired by fibration theory that decomposes trust-region maintenance into global and local components, providing unified multi-scale stability control for LLM policy optimization. To enhance architectural sparsity, the model comprises 48B total parameters while activating only 2.7B parameters per forward pass, achieving a substantially higher sparsity ratio than contemporary industry leading models of comparable scale. To further improve inference throughput, we adopt a joint training-inference co-design that incorporates dense Multi-Token Prediction (MTP) and Quantization-Aware Training (QAT). We release the checkpoints for both JoyAI-LLM-48B-A3B Base and its post-trained variants on Hugging Face to support the open-source community.
format	Preprint
id	arxiv_https___arxiv_org_abs_2604_03044
institution	arXiv
publishDate	2026
record_format	arxiv
spellingShingle	JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency Cai, Aichen Zhang, Anmeng Li, Anyu Zhang, Bo Cai, Bohua Li, Chang Jiang, Changjian Lu, Changkai Xue, Chao Liang, Chaocai Zhang, Cheng Liu, Dongkai Wang, Fei Huang, Guoqiang Ke, Haijian Lin, Han Wang, Hao Miao, Ji Zhang, Jiacheng Shi, Jialong Zhu, Jifeng Qian, Jingjing Luo, Junhui Xiong, Junwu So, Lam Huang, Liang Ke, Ming Li, Mingyang Shi, Panfeng Hao, Peng Wang, Qi Lai, Qian Yuan, Qiaoqiao Yin, Qingyu Cao, Qiong Wang, Qixiang Bian, Rongcheng Han, Rongduo Zheng, Shaoqiang Hu, Shi Suo, Shi Ren, Shijie Zhang, Shijin Fan, Shiying Xie, Shuai Zhang, Tianyi Liu, Wei Tan, Wentao Meng, Xianghan He, Xiaodong Pan, Xing Wang, Xiran Peng, Xuyang Zhang, Ya Liu, Yang Duan, Yangyang Chen, Yanxu Gong, Yicheng Huang, Yidan Liu, Yifei Bai, Yinhao Liu, Yongqiang Zhang, Yuesong Zhang, Yuqi Xie, Zerui Wang, Zhenfang Shen, Zhennan Liu, Zheyuan Zeng, Zhuwei Computation and Language Artificial Intelligence We introduce JoyAI-LLM Flash, an efficient Mixture-of-Experts (MoE) language model designed to redefine the trade-off between strong performance and token efficiency in the sub-50B parameter regime. JoyAI-LLM Flash is pretrained on a massive corpus of 20 trillion tokens and further optimized through a rigorous post-training pipeline, including supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and large-scale reinforcement learning (RL) across diverse environments. To improve token efficiency, JoyAI-LLM Flash strategically balances \emph{thinking} and \emph{non-thinking} cognitive modes and introduces FiberPO, a novel RL algorithm inspired by fibration theory that decomposes trust-region maintenance into global and local components, providing unified multi-scale stability control for LLM policy optimization. To enhance architectural sparsity, the model comprises 48B total parameters while activating only 2.7B parameters per forward pass, achieving a substantially higher sparsity ratio than contemporary industry leading models of comparable scale. To further improve inference throughput, we adopt a joint training-inference co-design that incorporates dense Multi-Token Prediction (MTP) and Quantization-Aware Training (QAT). We release the checkpoints for both JoyAI-LLM-48B-A3B Base and its post-trained variants on Hugging Face to support the open-source community.
title	JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency
topic	Computation and Language Artificial Intelligence
url	https://arxiv.org/abs/2604.03044