Learning to Plan Long-Term for Language Modeling
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910585897091072 |
|---|---|
| author | Mai, Florian Cornille, Nathan Moens, Marie-Francine |
| author_facet | Mai, Florian Cornille, Nathan Moens, Marie-Francine |
| contents | Modern language models predict the next token in the sequence by considering the past text through a powerful function such as attention. However, language models have no explicit mechanism that allows them to spend computation time for planning long-distance future text, leading to a suboptimal token prediction. In this paper, we propose a planner that predicts a latent plan for many sentences into the future. By sampling multiple plans at once, we condition the language model on an accurate approximation of the distribution of text continuations, which leads to better next token prediction accuracy. In effect, this allows trading computation time for prediction accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_00070 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning to Plan Long-Term for Language Modeling Mai, Florian Cornille, Nathan Moens, Marie-Francine Computation and Language Artificial Intelligence Modern language models predict the next token in the sequence by considering the past text through a powerful function such as attention. However, language models have no explicit mechanism that allows them to spend computation time for planning long-distance future text, leading to a suboptimal token prediction. In this paper, we propose a planner that predicts a latent plan for many sentences into the future. By sampling multiple plans at once, we condition the language model on an accurate approximation of the distribution of text continuations, which leads to better next token prediction accuracy. In effect, this allows trading computation time for prediction accuracy. |
| title | Learning to Plan Long-Term for Language Modeling |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2409.00070 |