Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911500393775104 |
|---|---|
| author | Lidin, Joel Sarfi, Amir Miahi, Erfan Anthony, Quentin Chauhan, Shivam Pappas, Evangelos Thérien, Benjamin Belilovsky, Eugene Dare, Samuel |
| author_facet | Lidin, Joel Sarfi, Amir Miahi, Erfan Anthony, Quentin Chauhan, Shivam Pappas, Evangelos Thérien, Benjamin Belilovsky, Eugene Dare, Samuel |
| contents | Recently, there has been increased interest in globally distributed training, which has the promise to both reduce training costs and democratize participation in building large-scale foundation models. However, existing models trained in a globally distributed manner are relatively small in scale and have only been trained with whitelisted participants. Therefore, they do not yet realize the full promise of democratized participation. In this report, we describe Covenant-72B, an LLM produced by the largest collaborative globally distributed pre-training run (in terms of both compute and model scale), which simultaneously allowed open, permissionless participation supported by a live blockchain protocol. We utilized a state-of-the-art communication-efficient optimizer, SparseLoCo, supporting dynamic participation with peers joining and leaving freely. Our model, pre-trained on approximately 1.1T tokens, performs competitively with fully centralized models pre-trained on similar or higher compute budgets, demonstrating that fully democratized, non-whitelisted participation is not only feasible, but can be achieved at unprecedented scale for a globally distributed pre-training run. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_08163 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet Lidin, Joel Sarfi, Amir Miahi, Erfan Anthony, Quentin Chauhan, Shivam Pappas, Evangelos Thérien, Benjamin Belilovsky, Eugene Dare, Samuel Distributed, Parallel, and Cluster Computing Machine Learning Recently, there has been increased interest in globally distributed training, which has the promise to both reduce training costs and democratize participation in building large-scale foundation models. However, existing models trained in a globally distributed manner are relatively small in scale and have only been trained with whitelisted participants. Therefore, they do not yet realize the full promise of democratized participation. In this report, we describe Covenant-72B, an LLM produced by the largest collaborative globally distributed pre-training run (in terms of both compute and model scale), which simultaneously allowed open, permissionless participation supported by a live blockchain protocol. We utilized a state-of-the-art communication-efficient optimizer, SparseLoCo, supporting dynamic participation with peers joining and leaving freely. Our model, pre-trained on approximately 1.1T tokens, performs competitively with fully centralized models pre-trained on similar or higher compute budgets, demonstrating that fully democratized, non-whitelisted participation is not only feasible, but can be achieved at unprecedented scale for a globally distributed pre-training run. |
| title | Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2603.08163 |