Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911435244699648 |
|---|---|
| author | Liu, Yuliang Song, Yunchong Wang, Yixuan Ge, Kewen Lamb, Alex Guo, Qipeng Chen, Kai Zhou, Bowen Lin, Zhouhan |
| author_facet | Liu, Yuliang Song, Yunchong Wang, Yixuan Ge, Kewen Lamb, Alex Guo, Qipeng Chen, Kai Zhou, Bowen Lin, Zhouhan |
| contents | We propose Next Concept Prediction (NCP), a generative pretraining paradigm built on top of Next Token Prediction (NTP). NCP predicts discrete concepts that span multiple tokens, thereby forming a more challenging pretraining objective. Our model, ConceptLM, quantizes hidden states using Vector Quantization and constructs a concept vocabulary. It leverages both NCP and NTP to drive parameter updates and generates a concept to guide the generation of the following tokens. We train ConceptLM from scratch at scales ranging from 70M to 1.5B parameters with up to 300B training data, including Pythia and GPT-2 backbones. Results on 13 benchmarks show that NCP yields consistent performance gains over traditional token-level models. Furthermore, continual pretraining experiments on an 8B-parameter Llama model indicate that NCP can further improve an NTP-trained model. Our analysis suggests that NCP leads to more powerful language models by introducing a harder pretraining task, providing a promising path toward better language modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_08984 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models Liu, Yuliang Song, Yunchong Wang, Yixuan Ge, Kewen Lamb, Alex Guo, Qipeng Chen, Kai Zhou, Bowen Lin, Zhouhan Computation and Language Artificial Intelligence We propose Next Concept Prediction (NCP), a generative pretraining paradigm built on top of Next Token Prediction (NTP). NCP predicts discrete concepts that span multiple tokens, thereby forming a more challenging pretraining objective. Our model, ConceptLM, quantizes hidden states using Vector Quantization and constructs a concept vocabulary. It leverages both NCP and NTP to drive parameter updates and generates a concept to guide the generation of the following tokens. We train ConceptLM from scratch at scales ranging from 70M to 1.5B parameters with up to 300B training data, including Pythia and GPT-2 backbones. Results on 13 benchmarks show that NCP yields consistent performance gains over traditional token-level models. Furthermore, continual pretraining experiments on an 8B-parameter Llama model indicate that NCP can further improve an NTP-trained model. Our analysis suggests that NCP leads to more powerful language models by introducing a harder pretraining task, providing a promising path toward better language modeling. |
| title | Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2602.08984 |