Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2310.10638 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914845857677312 |
|---|---|
| author | Shi, Weijia Min, Sewon Lomeli, Maria Zhou, Chunting Li, Margaret Szilvasy, Gergely James, Rich Lin, Xi Victoria Smith, Noah A. Zettlemoyer, Luke Yih, Scott Lewis, Mike |
| author_facet | Shi, Weijia Min, Sewon Lomeli, Maria Zhou, Chunting Li, Margaret Szilvasy, Gergely James, Rich Lin, Xi Victoria Smith, Noah A. Zettlemoyer, Luke Yih, Scott Lewis, Mike |
| contents | Large language models (LMs) are currently trained to predict tokens given document prefixes, enabling them to directly perform long-form generation and prompting-style tasks which can be reduced to document completion. Existing pretraining pipelines train LMs by concatenating random sets of short documents to create input contexts but the prior documents provide no signal for predicting the next document. We instead present In-Context Pretraining, a new approach where language models are pretrained on a sequence of related documents, thereby explicitly encouraging them to read and reason across document boundaries. We can do In-Context Pretraining by simply changing the document ordering so that each context contains related documents, and directly applying existing pretraining pipelines. However, this document sorting problem is challenging. There are billions of documents and we would like the sort to maximize contextual similarity for every document without repeating any data. To do this, we introduce approximate algorithms for finding related documents with efficient nearest neighbor search and constructing coherent input contexts with a graph traversal algorithm. Our experiments show In-Context Pretraining offers a simple and scalable approach to significantly enhance LMs'performance: we see notable improvements in tasks that require more complex contextual reasoning, including in-context learning (+8%), reading comprehension (+15%), faithfulness to previous contexts (+16%), long-context reasoning (+5%), and retrieval augmentation (+9%). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_10638 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | In-context Pretraining: Language Modeling Beyond Document Boundaries Shi, Weijia Min, Sewon Lomeli, Maria Zhou, Chunting Li, Margaret Szilvasy, Gergely James, Rich Lin, Xi Victoria Smith, Noah A. Zettlemoyer, Luke Yih, Scott Lewis, Mike Computation and Language Artificial Intelligence Machine Learning Large language models (LMs) are currently trained to predict tokens given document prefixes, enabling them to directly perform long-form generation and prompting-style tasks which can be reduced to document completion. Existing pretraining pipelines train LMs by concatenating random sets of short documents to create input contexts but the prior documents provide no signal for predicting the next document. We instead present In-Context Pretraining, a new approach where language models are pretrained on a sequence of related documents, thereby explicitly encouraging them to read and reason across document boundaries. We can do In-Context Pretraining by simply changing the document ordering so that each context contains related documents, and directly applying existing pretraining pipelines. However, this document sorting problem is challenging. There are billions of documents and we would like the sort to maximize contextual similarity for every document without repeating any data. To do this, we introduce approximate algorithms for finding related documents with efficient nearest neighbor search and constructing coherent input contexts with a graph traversal algorithm. Our experiments show In-Context Pretraining offers a simple and scalable approach to significantly enhance LMs'performance: we see notable improvements in tasks that require more complex contextual reasoning, including in-context learning (+8%), reading comprehension (+15%), faithfulness to previous contexts (+16%), long-context reasoning (+5%), and retrieval augmentation (+9%). |
| title | In-context Pretraining: Language Modeling Beyond Document Boundaries |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2310.10638 |