Analysing The Impact of Sequence Composition on Language Model Pre-Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915149853491200 |
|---|---|
| author | Zhao, Yu Qu, Yuanbin Staniszewski, Konrad Tworkowski, Szymon Liu, Wei Miłoś, Piotr Wu, Yuxiang Minervini, Pasquale |
| author_facet | Zhao, Yu Qu, Yuanbin Staniszewski, Konrad Tworkowski, Szymon Liu, Wei Miłoś, Piotr Wu, Yuxiang Minervini, Pasquale |
| contents | Most language model pre-training frameworks concatenate multiple documents into fixed-length sequences and use causal masking to compute the likelihood of each token given its context; this strategy is widely adopted due to its simplicity and efficiency. However, to this day, the influence of the pre-training sequence composition strategy on the generalisation properties of the model remains under-explored. In this work, we find that applying causal masking can lead to the inclusion of distracting information from previous documents during pre-training, which negatively impacts the performance of the models on language modelling and downstream tasks. In intra-document causal masking, the likelihood of each token is only conditioned on the previous tokens in the same document, eliminating potential distracting information from previous documents and significantly improving performance. Furthermore, we find that concatenating related documents can reduce some potential distractions during pre-training, and our proposed efficient retrieval-based sequence construction method, BM25Chunk, can improve in-context learning (+11.6\%), knowledge memorisation (+9.8\%), and context utilisation (+7.2\%) abilities of language models without sacrificing efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_13991 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Analysing The Impact of Sequence Composition on Language Model Pre-Training Zhao, Yu Qu, Yuanbin Staniszewski, Konrad Tworkowski, Szymon Liu, Wei Miłoś, Piotr Wu, Yuxiang Minervini, Pasquale Computation and Language Most language model pre-training frameworks concatenate multiple documents into fixed-length sequences and use causal masking to compute the likelihood of each token given its context; this strategy is widely adopted due to its simplicity and efficiency. However, to this day, the influence of the pre-training sequence composition strategy on the generalisation properties of the model remains under-explored. In this work, we find that applying causal masking can lead to the inclusion of distracting information from previous documents during pre-training, which negatively impacts the performance of the models on language modelling and downstream tasks. In intra-document causal masking, the likelihood of each token is only conditioned on the previous tokens in the same document, eliminating potential distracting information from previous documents and significantly improving performance. Furthermore, we find that concatenating related documents can reduce some potential distractions during pre-training, and our proposed efficient retrieval-based sequence construction method, BM25Chunk, can improve in-context learning (+11.6\%), knowledge memorisation (+9.8\%), and context utilisation (+7.2\%) abilities of language models without sacrificing efficiency. |
| title | Analysing The Impact of Sequence Composition on Language Model Pre-Training |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.13991 |