Context-level Language Modeling by Learning Predictive Context Embeddings
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918332729393152 |
|---|---|
| author | Dai, Beiya Liu, Yuliang Xue, Daozheng Song, Yunchong Guo, Qipeng Chen, Kai Wang, Xinbing Zhou, Bowen Lin, Zhouhan |
| author_facet | Dai, Beiya Liu, Yuliang Xue, Daozheng Song, Yunchong Guo, Qipeng Chen, Kai Wang, Xinbing Zhou, Bowen Lin, Zhouhan |
| contents | We propose ContextLM, a framework that implicitly learns multi-token prediction by augmenting standard pretraining with an intrinsic next-context prediction objective. ContextLM builds a language model on top of context embeddings that span multiple tokens, enabling better next-token prediction by predicting the next context. Our model is fully compatible with standard autoregressive, token-by-token evaluation paradigms (e.g., perplexity). Extensive experiments with GPT-2 and Pythia backbones (up to 1.5B parameters and 300B training tokens) reveal that ContextLM shifts the Pareto frontier of scaling laws, exhibiting superior efficiency in parameters, training tokens, and FLOPs. Our results show that ContextLM could already achieve the baseline perplexity using 39\% fewer parameters and demonstrates robust generalization improvements on extensive downstream tasks under equivalent parameter counts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_20280 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Context-level Language Modeling by Learning Predictive Context Embeddings Dai, Beiya Liu, Yuliang Xue, Daozheng Song, Yunchong Guo, Qipeng Chen, Kai Wang, Xinbing Zhou, Bowen Lin, Zhouhan Computation and Language Artificial Intelligence We propose ContextLM, a framework that implicitly learns multi-token prediction by augmenting standard pretraining with an intrinsic next-context prediction objective. ContextLM builds a language model on top of context embeddings that span multiple tokens, enabling better next-token prediction by predicting the next context. Our model is fully compatible with standard autoregressive, token-by-token evaluation paradigms (e.g., perplexity). Extensive experiments with GPT-2 and Pythia backbones (up to 1.5B parameters and 300B training tokens) reveal that ContextLM shifts the Pareto frontier of scaling laws, exhibiting superior efficiency in parameters, training tokens, and FLOPs. Our results show that ContextLM could already achieve the baseline perplexity using 39\% fewer parameters and demonstrates robust generalization improvements on extensive downstream tasks under equivalent parameter counts. |
| title | Context-level Language Modeling by Learning Predictive Context Embeddings |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.20280 |