SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908767856099328 |
|---|---|
| author | Ohib, Riyasat Thapaliya, Bishal Dziugaite, Gintare Karolina Liu, Jingyu Calhoun, Vince Plis, Sergey |
| author_facet | Ohib, Riyasat Thapaliya, Bishal Dziugaite, Gintare Karolina Liu, Jingyu Calhoun, Vince Plis, Sergey |
| contents | In this work, we propose Salient Sparse Federated Learning (SSFL), a streamlined approach for sparse federated learning with efficient communication. SSFL identifies a sparse subnetwork prior to training, leveraging parameter saliency scores computed separately on local client data in non-IID scenarios, and then aggregated, to determine a global mask. Only the sparse model weights are trained and communicated each round between the clients and the server. On standard benchmarks including CIFAR-10, CIFAR-100, and Tiny-ImageNet, SSFL consistently improves the accuracy sparsity trade off, achieving more than 20\% relative error reduction on CIFAR-10 compared to the strongest sparse baseline, while reducing communication costs by $2 \times$ relative to dense FL. Finally, in a real-world federated learning deployment, SSFL delivers over $2.3 \times$ faster communication time, underscoring its practical efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_09037 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learning Ohib, Riyasat Thapaliya, Bishal Dziugaite, Gintare Karolina Liu, Jingyu Calhoun, Vince Plis, Sergey Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing In this work, we propose Salient Sparse Federated Learning (SSFL), a streamlined approach for sparse federated learning with efficient communication. SSFL identifies a sparse subnetwork prior to training, leveraging parameter saliency scores computed separately on local client data in non-IID scenarios, and then aggregated, to determine a global mask. Only the sparse model weights are trained and communicated each round between the clients and the server. On standard benchmarks including CIFAR-10, CIFAR-100, and Tiny-ImageNet, SSFL consistently improves the accuracy sparsity trade off, achieving more than 20\% relative error reduction on CIFAR-10 compared to the strongest sparse baseline, while reducing communication costs by $2 \times$ relative to dense FL. Finally, in a real-world federated learning deployment, SSFL delivers over $2.3 \times$ faster communication time, underscoring its practical efficiency. |
| title | SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learning |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2405.09037 |