Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910069873967104 |
|---|---|
| author | Mahajan, Divyat Goyal, Sachin Idrissi, Badr Youbi Pezeshki, Mohammad Mitliagkas, Ioannis Lopez-Paz, David Ahuja, Kartik |
| author_facet | Mahajan, Divyat Goyal, Sachin Idrissi, Badr Youbi Pezeshki, Mohammad Mitliagkas, Ioannis Lopez-Paz, David Ahuja, Kartik |
| contents | Next-token prediction (NTP) has driven the success of large language models (LLMs), but it struggles with long-horizon reasoning, planning, and creative writing, with these limitations largely attributed to teacher-forced training. Multi-token prediction (MTP) partially mitigates these issues by predicting several future tokens at once, but it mostly captures short-range dependencies and offers limited improvement. We propose future summary prediction (FSP), which trains an auxiliary head to predict a compact representation of the long-term future, preserving information relevant for long-form generations. We explore two variants of FSP: handcrafted summaries, for example, a bag of words summary of the future sequence, and learned summaries, which use embeddings produced by a reverse language model trained from right-to-left order. Large-scale pretraining experiments (3B and 8B-parameter models) demonstrate that FSP provides improvements over both NTP and MTP across math, reasoning, and coding benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_14751 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries Mahajan, Divyat Goyal, Sachin Idrissi, Badr Youbi Pezeshki, Mohammad Mitliagkas, Ioannis Lopez-Paz, David Ahuja, Kartik Machine Learning Artificial Intelligence Next-token prediction (NTP) has driven the success of large language models (LLMs), but it struggles with long-horizon reasoning, planning, and creative writing, with these limitations largely attributed to teacher-forced training. Multi-token prediction (MTP) partially mitigates these issues by predicting several future tokens at once, but it mostly captures short-range dependencies and offers limited improvement. We propose future summary prediction (FSP), which trains an auxiliary head to predict a compact representation of the long-term future, preserving information relevant for long-form generations. We explore two variants of FSP: handcrafted summaries, for example, a bag of words summary of the future sequence, and learned summaries, which use embeddings produced by a reverse language model trained from right-to-left order. Large-scale pretraining experiments (3B and 8B-parameter models) demonstrate that FSP provides improvements over both NTP and MTP across math, reasoning, and coding benchmarks. |
| title | Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.14751 |