Gespeichert in:
| 1. Verfasser: | |
|---|---|
| Format: | Recurso digital |
| Sprache: | Englisch |
| Veröffentlicht: |
Zenodo
2026
|
| Online-Zugang: | https://doi.org/10.5281/zenodo.18130207 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866901537630978048 |
|---|---|
| author | Momen Ghazouani |
| author_facet | Momen Ghazouani |
| contents | <p>The training of large-scale neural networks increasingly suffers from communication bottlenecks across distributed accelerators, with gradient synchronization dominating both bandwidth consumption and energy costs. Conventional parameter exchange protocols operate under a strictly explicit transmission paradigm: every update, regardless of magnitude or novelty, is broadcast in full across the cluster. Empirical profiling of modern large language model (LLM) training reveals that 60–75% of gradient shards exhibit high temporal redundancy changes that fall below optimizer-effective thresholds or maintain near-identical values across consecutive iterations. Yet current all-reduce pipelines expend identical bandwidth on both critical and negligible updates, leading to disproportionate energy drain and interconnect congestion We introduce the Silent Gradient Protocol (SGP), a silence-aware training communication framework that reframes absence of signal as a valid synchronization state. Unlike traditional collectives that equate non-transmission with failure, SGP establishes a formal contract where intentional silence denotes that the accumulated gradient update is statistically negligible. Consequently, the receiver interprets silence as a zero-vector update, while the sender locally preserves the residual information for future iterations By elevating silence to a first-class protocol primitive, SGP eliminates redundant gradient traffic while maintaining convergence guarantees. Our deterministic distributed Simulation demonstrates that SGP achieves 90–95% communication reduction (Global Compression Ratio) while preserving model convergence within ±0.15% of dense baseline accuracy, effectively decoupling communication cost from model size in distributed neural training </p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_18130207 |
| institution | Zenodo |
| language | eng |
| publishDate | 2026 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Silence Is All We Need Momen Ghazouani <p>The training of large-scale neural networks increasingly suffers from communication bottlenecks across distributed accelerators, with gradient synchronization dominating both bandwidth consumption and energy costs. Conventional parameter exchange protocols operate under a strictly explicit transmission paradigm: every update, regardless of magnitude or novelty, is broadcast in full across the cluster. Empirical profiling of modern large language model (LLM) training reveals that 60–75% of gradient shards exhibit high temporal redundancy changes that fall below optimizer-effective thresholds or maintain near-identical values across consecutive iterations. Yet current all-reduce pipelines expend identical bandwidth on both critical and negligible updates, leading to disproportionate energy drain and interconnect congestion We introduce the Silent Gradient Protocol (SGP), a silence-aware training communication framework that reframes absence of signal as a valid synchronization state. Unlike traditional collectives that equate non-transmission with failure, SGP establishes a formal contract where intentional silence denotes that the accumulated gradient update is statistically negligible. Consequently, the receiver interprets silence as a zero-vector update, while the sender locally preserves the residual information for future iterations By elevating silence to a first-class protocol primitive, SGP eliminates redundant gradient traffic while maintaining convergence guarantees. Our deterministic distributed Simulation demonstrates that SGP achieves 90–95% communication reduction (Global Compression Ratio) while preserving model convergence within ±0.15% of dense baseline accuracy, effectively decoupling communication cost from model size in distributed neural training </p> |
| title | Silence Is All We Need |
| url | https://doi.org/10.5281/zenodo.18130207 |