Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2502.08524 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929712076423168 |
|---|---|
| author | Tack, Jihoon Lanchantin, Jack Yu, Jane Cohen, Andrew Kulikov, Ilia Lan, Janice Hao, Shibo Tian, Yuandong Weston, Jason Li, Xian |
| author_facet | Tack, Jihoon Lanchantin, Jack Yu, Jane Cohen, Andrew Kulikov, Ilia Lan, Janice Hao, Shibo Tian, Yuandong Weston, Jason Li, Xian |
| contents | Next token prediction has been the standard training objective used in large language model pretraining. Representations are learned as a result of optimizing for token-level perplexity. We propose Continuous Concept Mixing (CoCoMix), a novel pretraining framework that combines discrete next token prediction with continuous concepts. Specifically, CoCoMix predicts continuous concepts learned from a pretrained sparse autoencoder and mixes them into the model's hidden state by interleaving with token hidden representations. Through experiments on multiple benchmarks, including language modeling and downstream reasoning tasks, we show that CoCoMix is more sample efficient and consistently outperforms standard next token prediction, knowledge distillation and inserting pause tokens. We find that combining both concept learning and interleaving in an end-to-end framework is critical to performance gains. Furthermore, CoCoMix enhances interpretability and steerability by allowing direct inspection and modification of the predicted concept, offering a transparent way to guide the model's internal reasoning process. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_08524 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LLM Pretraining with Continuous Concepts Tack, Jihoon Lanchantin, Jack Yu, Jane Cohen, Andrew Kulikov, Ilia Lan, Janice Hao, Shibo Tian, Yuandong Weston, Jason Li, Xian Machine Learning Computation and Language Next token prediction has been the standard training objective used in large language model pretraining. Representations are learned as a result of optimizing for token-level perplexity. We propose Continuous Concept Mixing (CoCoMix), a novel pretraining framework that combines discrete next token prediction with continuous concepts. Specifically, CoCoMix predicts continuous concepts learned from a pretrained sparse autoencoder and mixes them into the model's hidden state by interleaving with token hidden representations. Through experiments on multiple benchmarks, including language modeling and downstream reasoning tasks, we show that CoCoMix is more sample efficient and consistently outperforms standard next token prediction, knowledge distillation and inserting pause tokens. We find that combining both concept learning and interleaving in an end-to-end framework is critical to performance gains. Furthermore, CoCoMix enhances interpretability and steerability by allowing direct inspection and modification of the predicted concept, offering a transparent way to guide the model's internal reasoning process. |
| title | LLM Pretraining with Continuous Concepts |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2502.08524 |