Scaling Embeddings Outperforms Scaling Experts in Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914320598695936 |
|---|---|
| author | Liu, Hong Zhang, Jiaqi Wang, Chao Hu, Xing Lyu, Linkun Sun, Jiaqi Yang, Xurui Wang, Bo Li, Fengcun Qian, Yulei Si, Lingtong Sun, Yerui Li, Rumei Pei, Peng Xie, Yuchen Cai, Xunliang |
| author_facet | Liu, Hong Zhang, Jiaqi Wang, Chao Hu, Xing Lyu, Linkun Sun, Jiaqi Yang, Xurui Wang, Bo Li, Fengcun Qian, Yulei Si, Lingtong Sun, Yerui Li, Rumei Pei, Peng Xie, Yuchen Cai, Xunliang |
| contents | While Mixture-of-Experts (MoE) architectures have become the standard for sparsity scaling in large language models, they increasingly face diminishing returns and system-level bottlenecks. In this work, we explore embedding scaling as a potent, orthogonal dimension for scaling sparsity. Through a comprehensive analysis and experiments, we identify specific regimes where embedding scaling achieves a superior Pareto frontier compared to expert scaling. We systematically characterize the critical architectural factors governing this efficacy -- ranging from parameter budgeting to the interplay with model width and depth. Moreover, by integrating tailored system optimizations and speculative decoding, we effectively convert this sparsity into tangible inference speedups. Guided by these insights, we introduce LongCat-Flash-Lite, a 68.5B parameter model with ~3B activated trained from scratch. Despite allocating over 30B parameters to embeddings, LongCat-Flash-Lite not only surpasses parameter-equivalent MoE baselines but also exhibits exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_21204 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Scaling Embeddings Outperforms Scaling Experts in Language Models Liu, Hong Zhang, Jiaqi Wang, Chao Hu, Xing Lyu, Linkun Sun, Jiaqi Yang, Xurui Wang, Bo Li, Fengcun Qian, Yulei Si, Lingtong Sun, Yerui Li, Rumei Pei, Peng Xie, Yuchen Cai, Xunliang Computation and Language Artificial Intelligence Machine Learning While Mixture-of-Experts (MoE) architectures have become the standard for sparsity scaling in large language models, they increasingly face diminishing returns and system-level bottlenecks. In this work, we explore embedding scaling as a potent, orthogonal dimension for scaling sparsity. Through a comprehensive analysis and experiments, we identify specific regimes where embedding scaling achieves a superior Pareto frontier compared to expert scaling. We systematically characterize the critical architectural factors governing this efficacy -- ranging from parameter budgeting to the interplay with model width and depth. Moreover, by integrating tailored system optimizations and speculative decoding, we effectively convert this sparsity into tangible inference speedups. Guided by these insights, we introduce LongCat-Flash-Lite, a 68.5B parameter model with ~3B activated trained from scratch. Despite allocating over 30B parameters to embeddings, LongCat-Flash-Lite not only surpasses parameter-equivalent MoE baselines but also exhibits exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains. |
| title | Scaling Embeddings Outperforms Scaling Experts in Language Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2601.21204 |