Recurrent Early Exits for Federated Learning with Heterogeneous Clients
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929359012495360 |
|---|---|
| author | Lee, Royson Fernandez-Marques, Javier Hu, Shell Xu Li, Da Laskaridis, Stefanos Dudziak, Łukasz Hospedales, Timothy Huszár, Ferenc Lane, Nicholas D. |
| author_facet | Lee, Royson Fernandez-Marques, Javier Hu, Shell Xu Li, Da Laskaridis, Stefanos Dudziak, Łukasz Hospedales, Timothy Huszár, Ferenc Lane, Nicholas D. |
| contents | Federated learning (FL) has enabled distributed learning of a model across multiple clients in a privacy-preserving manner. One of the main challenges of FL is to accommodate clients with varying hardware capacities; clients have differing compute and memory requirements. To tackle this challenge, recent state-of-the-art approaches leverage the use of early exits. Nonetheless, these approaches fall short of mitigating the challenges of joint learning multiple exit classifiers, often relying on hand-picked heuristic solutions for knowledge distillation among classifiers and/or utilizing additional layers for weaker classifiers. In this work, instead of utilizing multiple classifiers, we propose a recurrent early exit approach named ReeFL that fuses features from different sub-models into a single shared classifier. Specifically, we use a transformer-based early-exit module shared among sub-models to i) better exploit multi-layer feature representations for task-specific prediction and ii) modulate the feature representation of the backbone model for subsequent predictions. We additionally present a per-client self-distillation approach where the best sub-model is automatically selected as the teacher of the other sub-models at each client. Our experiments on standard image and speech classification benchmarks across various emerging federated fine-tuning baselines demonstrate ReeFL's effectiveness over previous works. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_14791 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Recurrent Early Exits for Federated Learning with Heterogeneous Clients Lee, Royson Fernandez-Marques, Javier Hu, Shell Xu Li, Da Laskaridis, Stefanos Dudziak, Łukasz Hospedales, Timothy Huszár, Ferenc Lane, Nicholas D. Machine Learning Computer Vision and Pattern Recognition Distributed, Parallel, and Cluster Computing Federated learning (FL) has enabled distributed learning of a model across multiple clients in a privacy-preserving manner. One of the main challenges of FL is to accommodate clients with varying hardware capacities; clients have differing compute and memory requirements. To tackle this challenge, recent state-of-the-art approaches leverage the use of early exits. Nonetheless, these approaches fall short of mitigating the challenges of joint learning multiple exit classifiers, often relying on hand-picked heuristic solutions for knowledge distillation among classifiers and/or utilizing additional layers for weaker classifiers. In this work, instead of utilizing multiple classifiers, we propose a recurrent early exit approach named ReeFL that fuses features from different sub-models into a single shared classifier. Specifically, we use a transformer-based early-exit module shared among sub-models to i) better exploit multi-layer feature representations for task-specific prediction and ii) modulate the feature representation of the backbone model for subsequent predictions. We additionally present a per-client self-distillation approach where the best sub-model is automatically selected as the teacher of the other sub-models at each client. Our experiments on standard image and speech classification benchmarks across various emerging federated fine-tuning baselines demonstrate ReeFL's effectiveness over previous works. |
| title | Recurrent Early Exits for Federated Learning with Heterogeneous Clients |
| topic | Machine Learning Computer Vision and Pattern Recognition Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2405.14791 |