Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.04333 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918485798420480 |
|---|---|
| author | Araujo, Joao Chow, Alex Handley, Mark Lewis, Ryder Paasch, Christoph Padhye, Jitendra Papamichael, Michael Steinbrecher, Greg Tootoonchian, Amin Yuan, Lihua Anantharamu, S. Dosi, Abhishek Garg, Mohit Ghazi, Mahdieh Hoefler, Torsten Jayasinghe, Deepal Jose, Jithin Kabbani, Abdul Lu, Guohan Wang, Yang Doddapaneni, K. Garimella, Murali Jain, Vipin Le, Yanfang Nagulapalli, H. Narayanan, S. Pan, Rong Sabesan, Rathina Sivaramu, Raghava Sohan, Rip Davis, Eric Dumitrescu, Dragos Kalkunte, Mohan Mitra, Bhaswar Morandin, Guglielmo Popa, Adrian Raiciu, Costin Spada, Eric Spillane, John Vaidya, Niranjan Barnea, Aviv Burstein, Idan Cohen, Elazar Friedman, Yamin Katz, Noam Moshref, Masoud Shpigelman, Yuval Shuler, Shahaf Shyman, Shy Sur, Sayantan |
| author_facet | Araujo, Joao Chow, Alex Handley, Mark Lewis, Ryder Paasch, Christoph Padhye, Jitendra Papamichael, Michael Steinbrecher, Greg Tootoonchian, Amin Yuan, Lihua Anantharamu, S. Dosi, Abhishek Garg, Mohit Ghazi, Mahdieh Hoefler, Torsten Jayasinghe, Deepal Jose, Jithin Kabbani, Abdul Lu, Guohan Wang, Yang Doddapaneni, K. Garimella, Murali Jain, Vipin Le, Yanfang Nagulapalli, H. Narayanan, S. Pan, Rong Sabesan, Rathina Sivaramu, Raghava Sohan, Rip Davis, Eric Dumitrescu, Dragos Kalkunte, Mohan Mitra, Bhaswar Morandin, Guglielmo Popa, Adrian Raiciu, Costin Spada, Eric Spillane, John Vaidya, Niranjan Barnea, Aviv Burstein, Idan Cohen, Elazar Friedman, Yamin Katz, Noam Moshref, Masoud Shpigelman, Yuval Shuler, Shahaf Shyman, Shy Sur, Sayantan |
| contents | Tail latency dominates the performance of synchronous pretraining jobs when running at very large scales. We describe a three-pronged approach: (1) a new RDMA-based transport protocol, MRC, sprays across many paths and actively load-balances between them, eliminating the issue of flow collisions (2) the use of multi-plane Clos topologies to get the benefits of high switch radix and redundancy, allowing training clusters well over 100K GPUs to be built as two-tier topologies while increasing physical redundancy, and (3) the use of static source-routing using SRv6 to allow MRC the freedom to bypass failures by itself. We describe our experiences running MRC and static SRv6 routing in production in OpenAI and Microsoft's largest training clusters, where it has been used to train the latest frontier models. We demonstrate how MRC allows AI training jobs to ride out many network failures that previously would have interrupted training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_04333 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Resilient AI Supercomputer Networking using MRC and SRv6 Araujo, Joao Chow, Alex Handley, Mark Lewis, Ryder Paasch, Christoph Padhye, Jitendra Papamichael, Michael Steinbrecher, Greg Tootoonchian, Amin Yuan, Lihua Anantharamu, S. Dosi, Abhishek Garg, Mohit Ghazi, Mahdieh Hoefler, Torsten Jayasinghe, Deepal Jose, Jithin Kabbani, Abdul Lu, Guohan Wang, Yang Doddapaneni, K. Garimella, Murali Jain, Vipin Le, Yanfang Nagulapalli, H. Narayanan, S. Pan, Rong Sabesan, Rathina Sivaramu, Raghava Sohan, Rip Davis, Eric Dumitrescu, Dragos Kalkunte, Mohan Mitra, Bhaswar Morandin, Guglielmo Popa, Adrian Raiciu, Costin Spada, Eric Spillane, John Vaidya, Niranjan Barnea, Aviv Burstein, Idan Cohen, Elazar Friedman, Yamin Katz, Noam Moshref, Masoud Shpigelman, Yuval Shuler, Shahaf Shyman, Shy Sur, Sayantan Networking and Internet Architecture Artificial Intelligence Distributed, Parallel, and Cluster Computing C.2.2; I.2 Tail latency dominates the performance of synchronous pretraining jobs when running at very large scales. We describe a three-pronged approach: (1) a new RDMA-based transport protocol, MRC, sprays across many paths and actively load-balances between them, eliminating the issue of flow collisions (2) the use of multi-plane Clos topologies to get the benefits of high switch radix and redundancy, allowing training clusters well over 100K GPUs to be built as two-tier topologies while increasing physical redundancy, and (3) the use of static source-routing using SRv6 to allow MRC the freedom to bypass failures by itself. We describe our experiences running MRC and static SRv6 routing in production in OpenAI and Microsoft's largest training clusters, where it has been used to train the latest frontier models. We demonstrate how MRC allows AI training jobs to ride out many network failures that previously would have interrupted training. |
| title | Resilient AI Supercomputer Networking using MRC and SRv6 |
| topic | Networking and Internet Architecture Artificial Intelligence Distributed, Parallel, and Cluster Computing C.2.2; I.2 |
| url | https://arxiv.org/abs/2605.04333 |