ProFe: Communication-Efficient Decentralized Federated Learning via Distillation and Prototypes
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916525048332288 |
|---|---|
| author | Sánchez, Pedro Miguel Sánchez Beltrán, Enrique Tomás Martínez Llamas, Miguel Fernández Bovet, Gérôme Pérez, Gregorio Martínez Celdrán, Alberto Huertas |
| author_facet | Sánchez, Pedro Miguel Sánchez Beltrán, Enrique Tomás Martínez Llamas, Miguel Fernández Bovet, Gérôme Pérez, Gregorio Martínez Celdrán, Alberto Huertas |
| contents | Decentralized Federated Learning (DFL) trains models in a collaborative and privacy-preserving manner while removing model centralization risks and improving communication bottlenecks. However, DFL faces challenges in efficient communication management and model aggregation within decentralized environments, especially with heterogeneous data distributions. Thus, this paper introduces ProFe, a novel communication optimization algorithm for DFL that combines knowledge distillation, prototype learning, and quantization techniques. ProFe utilizes knowledge from large local models to train smaller ones for aggregation, incorporates prototypes to better learn unseen classes, and applies quantization to reduce data transmitted during communication rounds. The performance of ProFe has been validated and compared to the literature by using benchmark datasets like MNIST, CIFAR10, and CIFAR100. Results showed that the proposed algorithm reduces communication costs by up to ~40-50% while maintaining or improving model performance. In addition, it adds ~20% training time due to increased complexity, generating a trade-off. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_11207 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ProFe: Communication-Efficient Decentralized Federated Learning via Distillation and Prototypes Sánchez, Pedro Miguel Sánchez Beltrán, Enrique Tomás Martínez Llamas, Miguel Fernández Bovet, Gérôme Pérez, Gregorio Martínez Celdrán, Alberto Huertas Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Networking and Internet Architecture Decentralized Federated Learning (DFL) trains models in a collaborative and privacy-preserving manner while removing model centralization risks and improving communication bottlenecks. However, DFL faces challenges in efficient communication management and model aggregation within decentralized environments, especially with heterogeneous data distributions. Thus, this paper introduces ProFe, a novel communication optimization algorithm for DFL that combines knowledge distillation, prototype learning, and quantization techniques. ProFe utilizes knowledge from large local models to train smaller ones for aggregation, incorporates prototypes to better learn unseen classes, and applies quantization to reduce data transmitted during communication rounds. The performance of ProFe has been validated and compared to the literature by using benchmark datasets like MNIST, CIFAR10, and CIFAR100. Results showed that the proposed algorithm reduces communication costs by up to ~40-50% while maintaining or improving model performance. In addition, it adds ~20% training time due to increased complexity, generating a trade-off. |
| title | ProFe: Communication-Efficient Decentralized Federated Learning via Distillation and Prototypes |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Networking and Internet Architecture |
| url | https://arxiv.org/abs/2412.11207 |