Layer-Parallel Training for Transformers
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915752214265856 |
|---|---|
| author | Jiang, Shuai Salvadó-Benasco, Marc Cyr, Eric C. Kopaničáková, Alena Krause, Rolf Schroder, Jacob B. |
| author_facet | Jiang, Shuai Salvadó-Benasco, Marc Cyr, Eric C. Kopaničáková, Alena Krause, Rolf Schroder, Jacob B. |
| contents | We present a new training methodology for transformers using a multilevel, layer-parallel approach. Through a neural ODE formulation of transformers, our application of a multilevel parallel-in-time algorithm for the forward and backpropagation phases of training achieves parallel acceleration over the layer dimension. This dramatically enhances parallel scalability as the network depth increases, which is particularly useful for increasingly large foundational models. However, achieving this introduces errors that cause systematic bias in the gradients, which in turn reduces convergence when closer to the minima. We develop an algorithm to detect this critical transition and either switch to serial training or systematically increase the accuracy of layer-parallel training. Results, including BERT, GPT2, ViT, and machine translation architectures, demonstrate parallel-acceleration as well as accuracy commensurate with serial pre-training while fine-tuning is unaffected. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_09026 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Layer-Parallel Training for Transformers Jiang, Shuai Salvadó-Benasco, Marc Cyr, Eric C. Kopaničáková, Alena Krause, Rolf Schroder, Jacob B. Machine Learning 68T07, 65Y05 I.2.6 We present a new training methodology for transformers using a multilevel, layer-parallel approach. Through a neural ODE formulation of transformers, our application of a multilevel parallel-in-time algorithm for the forward and backpropagation phases of training achieves parallel acceleration over the layer dimension. This dramatically enhances parallel scalability as the network depth increases, which is particularly useful for increasingly large foundational models. However, achieving this introduces errors that cause systematic bias in the gradients, which in turn reduces convergence when closer to the minima. We develop an algorithm to detect this critical transition and either switch to serial training or systematically increase the accuracy of layer-parallel training. Results, including BERT, GPT2, ViT, and machine translation architectures, demonstrate parallel-acceleration as well as accuracy commensurate with serial pre-training while fine-tuning is unaffected. |
| title | Layer-Parallel Training for Transformers |
| topic | Machine Learning 68T07, 65Y05 I.2.6 |
| url | https://arxiv.org/abs/2601.09026 |