Salvato in:
| Autore principale: | |
|---|---|
| Natura: | Recurso digital |
| Lingua: | inglese |
| Pubblicazione: |
Zenodo
2026
|
| Accesso online: | https://doi.org/10.5281/zenodo.18130207 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
Sommario:
- <p>The training of large-scale neural networks increasingly suffers from communication bottlenecks across distributed accelerators, with gradient synchronization dominating both bandwidth consumption and energy costs. Conventional parameter exchange protocols operate under a strictly explicit transmission paradigm: every update, regardless of magnitude or novelty, is broadcast in full across the cluster. Empirical profiling of modern large language model (LLM) training reveals that 60–75% of gradient shards exhibit high temporal redundancy changes that fall below optimizer-effective thresholds or maintain near-identical values across consecutive iterations. Yet current all-reduce pipelines expend identical bandwidth on both critical and negligible updates, leading to disproportionate energy drain and interconnect congestion We introduce the Silent Gradient Protocol (SGP), a silence-aware training communication framework that reframes absence of signal as a valid synchronization state. Unlike traditional collectives that equate non-transmission with failure, SGP establishes a formal contract where intentional silence denotes that the accumulated gradient update is statistically negligible. Consequently, the receiver interprets silence as a zero-vector update, while the sender locally preserves the residual information for future iterations By elevating silence to a first-class protocol primitive, SGP eliminates redundant gradient traffic while maintaining convergence guarantees. Our deterministic distributed Simulation demonstrates that SGP achieves 90–95% communication reduction (Global Compression Ratio) while preserving model convergence within ±0.15% of dense baseline accuracy, effectively decoupling communication cost from model size in distributed neural training </p>