Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.24617 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915706498449408 |
|---|---|
| author | Qu, Xingwei Wang, Shaowen Huang, Zihao Hua, Kai Yin, Fan Zhu, Rui-Jie Zhou, Jundong Min, Qiyang Wang, Zihao Li, Yizhi Zhang, Tianyu Xing, He Zhang, Zheng Song, Yuxuan Zheng, Tianyu Zeng, Zhiyuan Lin, Chenghua Zhang, Ge Huang, Wenhao |
| author_facet | Qu, Xingwei Wang, Shaowen Huang, Zihao Hua, Kai Yin, Fan Zhu, Rui-Jie Zhou, Jundong Min, Qiyang Wang, Zihao Li, Yizhi Zhang, Tianyu Xing, He Zhang, Zheng Song, Yuxuan Zheng, Tianyu Zeng, Zhiyuan Lin, Chenghua Zhang, Ge Huang, Wenhao |
| contents | Large Language Models (LLMs) apply uniform computation to all tokens, despite language exhibiting highly non-uniform information density. This token-uniform regime wastes capacity on locally predictable spans while under-allocating computation to semantically critical transitions. We propose $\textbf{Dynamic Large Concept Models (DLCM)}$, a hierarchical language modeling framework that learns semantic boundaries from latent representations and shifts computation from tokens to a compressed concept space where reasoning is more efficient. DLCM discovers variable-length concepts end-to-end without relying on predefined linguistic units. Hierarchical compression fundamentally changes scaling behavior. We introduce the first $\textbf{compression-aware scaling law}$, which disentangles token-level capacity, concept-level reasoning capacity, and compression ratio, enabling principled compute allocation under fixed FLOPs. To stably train this heterogeneous architecture, we further develop a $\textbf{decoupled $μ$P parametrization}$ that supports zero-shot hyperparameter transfer across widths and compression regimes. At a practical setting ($R=4$, corresponding to an average of four tokens per concept), DLCM reallocates roughly one-third of inference compute into a higher-capacity reasoning backbone, achieving a $\textbf{+2.69$\%$ average improvement}$ across 12 zero-shot benchmarks under matched inference FLOPs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_24617 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space Qu, Xingwei Wang, Shaowen Huang, Zihao Hua, Kai Yin, Fan Zhu, Rui-Jie Zhou, Jundong Min, Qiyang Wang, Zihao Li, Yizhi Zhang, Tianyu Xing, He Zhang, Zheng Song, Yuxuan Zheng, Tianyu Zeng, Zhiyuan Lin, Chenghua Zhang, Ge Huang, Wenhao Machine Learning Artificial Intelligence Large Language Models (LLMs) apply uniform computation to all tokens, despite language exhibiting highly non-uniform information density. This token-uniform regime wastes capacity on locally predictable spans while under-allocating computation to semantically critical transitions. We propose $\textbf{Dynamic Large Concept Models (DLCM)}$, a hierarchical language modeling framework that learns semantic boundaries from latent representations and shifts computation from tokens to a compressed concept space where reasoning is more efficient. DLCM discovers variable-length concepts end-to-end without relying on predefined linguistic units. Hierarchical compression fundamentally changes scaling behavior. We introduce the first $\textbf{compression-aware scaling law}$, which disentangles token-level capacity, concept-level reasoning capacity, and compression ratio, enabling principled compute allocation under fixed FLOPs. To stably train this heterogeneous architecture, we further develop a $\textbf{decoupled $μ$P parametrization}$ that supports zero-shot hyperparameter transfer across widths and compression regimes. At a practical setting ($R=4$, corresponding to an average of four tokens per concept), DLCM reallocates roughly one-third of inference compute into a higher-capacity reasoning backbone, achieving a $\textbf{+2.69$\%$ average improvement}$ across 12 zero-shot benchmarks under matched inference FLOPs. |
| title | Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2512.24617 |