SENTINEL: Stagewise Integrity Verification for Pipeline Parallel Decentralized Training
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914367939805184 |
|---|---|
| author | Dolatabadi, Hadi Mohaghegh Ajanthan, Thalaiyasingam Ramasinghe, Sameera Koneputugodage, Chamin P Hewa Avraham, Gil Zuo, Yan Shevchenko, Violetta Long, Alexander |
| author_facet | Dolatabadi, Hadi Mohaghegh Ajanthan, Thalaiyasingam Ramasinghe, Sameera Koneputugodage, Chamin P Hewa Avraham, Gil Zuo, Yan Shevchenko, Violetta Long, Alexander |
| contents | Decentralized training introduces critical security risks when executed across untrusted, geographically distributed nodes. While existing Byzantine-tolerant literature addresses data parallel (DP) training through robust aggregation methods, pipeline parallelism (PP) presents fundamentally distinct challenges. In PP, model layers are distributed across workers where the activations and their gradients flow between stages rather than being aggregated, making traditional DP approaches inapplicable. We propose SENTINEL, a verification mechanism for PP training without computation duplication. SENTINEL employs lightweight momentum-based monitoring using exponential moving averages (EMAs) to detect corrupted inter-stage communication. Unlike existing Byzantine-tolerant approaches for DP that aggregate parameter gradients across replicas, our approach verifies sequential activation/gradient transmission between layers. We provide theoretical convergence guarantees for this new setting that recovers classical convergence rates when relaxed to standard training. Experiments demonstrate successful training of up to 4B-parameter LLMs across untrusted distributed environments with up to 176 workers while maintaining model convergence and performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_03592 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SENTINEL: Stagewise Integrity Verification for Pipeline Parallel Decentralized Training Dolatabadi, Hadi Mohaghegh Ajanthan, Thalaiyasingam Ramasinghe, Sameera Koneputugodage, Chamin P Hewa Avraham, Gil Zuo, Yan Shevchenko, Violetta Long, Alexander Distributed, Parallel, and Cluster Computing Cryptography and Security Machine Learning Decentralized training introduces critical security risks when executed across untrusted, geographically distributed nodes. While existing Byzantine-tolerant literature addresses data parallel (DP) training through robust aggregation methods, pipeline parallelism (PP) presents fundamentally distinct challenges. In PP, model layers are distributed across workers where the activations and their gradients flow between stages rather than being aggregated, making traditional DP approaches inapplicable. We propose SENTINEL, a verification mechanism for PP training without computation duplication. SENTINEL employs lightweight momentum-based monitoring using exponential moving averages (EMAs) to detect corrupted inter-stage communication. Unlike existing Byzantine-tolerant approaches for DP that aggregate parameter gradients across replicas, our approach verifies sequential activation/gradient transmission between layers. We provide theoretical convergence guarantees for this new setting that recovers classical convergence rates when relaxed to standard training. Experiments demonstrate successful training of up to 4B-parameter LLMs across untrusted distributed environments with up to 176 workers while maintaining model convergence and performance. |
| title | SENTINEL: Stagewise Integrity Verification for Pipeline Parallel Decentralized Training |
| topic | Distributed, Parallel, and Cluster Computing Cryptography and Security Machine Learning |
| url | https://arxiv.org/abs/2603.03592 |