Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910475370889216 |
|---|---|
| author | Soldaini, Luca Kinney, Rodney Bhagia, Akshita Schwenk, Dustin Atkinson, David Authur, Russell Bogin, Ben Chandu, Khyathi Dumas, Jennifer Elazar, Yanai Hofmann, Valentin Jha, Ananya Harsh Kumar, Sachin Lucy, Li Lyu, Xinxi Lambert, Nathan Magnusson, Ian Morrison, Jacob Muennighoff, Niklas Naik, Aakanksha Nam, Crystal Peters, Matthew E. Ravichander, Abhilasha Richardson, Kyle Shen, Zejiang Strubell, Emma Subramani, Nishant Tafjord, Oyvind Walsh, Pete Zettlemoyer, Luke Smith, Noah A. Hajishirzi, Hannaneh Beltagy, Iz Groeneveld, Dirk Dodge, Jesse Lo, Kyle |
| author_facet | Soldaini, Luca Kinney, Rodney Bhagia, Akshita Schwenk, Dustin Atkinson, David Authur, Russell Bogin, Ben Chandu, Khyathi Dumas, Jennifer Elazar, Yanai Hofmann, Valentin Jha, Ananya Harsh Kumar, Sachin Lucy, Li Lyu, Xinxi Lambert, Nathan Magnusson, Ian Morrison, Jacob Muennighoff, Niklas Naik, Aakanksha Nam, Crystal Peters, Matthew E. Ravichander, Abhilasha Richardson, Kyle Shen, Zejiang Strubell, Emma Subramani, Nishant Tafjord, Oyvind Walsh, Pete Zettlemoyer, Luke Smith, Noah A. Hajishirzi, Hannaneh Beltagy, Iz Groeneveld, Dirk Dodge, Jesse Lo, Kyle |
| contents | Information about pretraining corpora used to train the current best-performing language models is seldom discussed: commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them. As a result, it is challenging to conduct and advance scientific research on language modeling, such as understanding how training data impacts model capabilities and limitations. To facilitate scientific research on language model pretraining, we curate and release Dolma, a three-trillion-token English corpus, built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials. We extensively document Dolma, including its design principles, details about its construction, and a summary of its contents. We present analyses and experimental results on intermediate states of Dolma to share what we have learned about important data curation practices. Finally, we open-source our data curation toolkit to enable reproduction of our work as well as support further research in large-scale data curation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_00159 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research Soldaini, Luca Kinney, Rodney Bhagia, Akshita Schwenk, Dustin Atkinson, David Authur, Russell Bogin, Ben Chandu, Khyathi Dumas, Jennifer Elazar, Yanai Hofmann, Valentin Jha, Ananya Harsh Kumar, Sachin Lucy, Li Lyu, Xinxi Lambert, Nathan Magnusson, Ian Morrison, Jacob Muennighoff, Niklas Naik, Aakanksha Nam, Crystal Peters, Matthew E. Ravichander, Abhilasha Richardson, Kyle Shen, Zejiang Strubell, Emma Subramani, Nishant Tafjord, Oyvind Walsh, Pete Zettlemoyer, Luke Smith, Noah A. Hajishirzi, Hannaneh Beltagy, Iz Groeneveld, Dirk Dodge, Jesse Lo, Kyle Computation and Language Information about pretraining corpora used to train the current best-performing language models is seldom discussed: commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them. As a result, it is challenging to conduct and advance scientific research on language modeling, such as understanding how training data impacts model capabilities and limitations. To facilitate scientific research on language model pretraining, we curate and release Dolma, a three-trillion-token English corpus, built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials. We extensively document Dolma, including its design principles, details about its construction, and a summary of its contents. We present analyses and experimental results on intermediate states of Dolma to share what we have learned about important data curation practices. Finally, we open-source our data curation toolkit to enable reproduction of our work as well as support further research in large-scale data curation. |
| title | Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.00159 |