FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communication
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911994970374144 |
|---|---|
| author | Yuan, Liangqi Han, Dong-Jun Chellapandi, Vishnu Pandi Żak, Stanislaw H. Brinton, Christopher G. |
| author_facet | Yuan, Liangqi Han, Dong-Jun Chellapandi, Vishnu Pandi Żak, Stanislaw H. Brinton, Christopher G. |
| contents | Multimodal federated learning (FL) aims to enrich model training in FL settings where devices are collecting measurements across multiple modalities (e.g., sensors measuring pressure, motion, and other types of data). However, key challenges to multimodal FL remain unaddressed, particularly in heterogeneous network settings: (i) the set of modalities collected by each device will be diverse, and (ii) communication limitations prevent devices from uploading all their locally trained modality models to the server. In this paper, we propose Federated Multimodal Fusion learning with Selective modality communication (FedMFS), a new multimodal fusion FL methodology that can tackle the above mentioned challenges. The key idea is the introduction of a modality selection criterion for each device, which weighs (i) the impact of the modality, gauged by Shapley value analysis, against (ii) the modality model size as a gauge for communication overhead. This enables FedMFS to flexibly balance performance against communication costs, depending on resource constraints and application requirements. Experiments on the real-world ActionSense dataset demonstrate the ability of FedMFS to achieve comparable accuracy to several baselines while reducing the communication overhead by over 4x. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_07048 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communication Yuan, Liangqi Han, Dong-Jun Chellapandi, Vishnu Pandi Żak, Stanislaw H. Brinton, Christopher G. Machine Learning Distributed, Parallel, and Cluster Computing Networking and Internet Architecture Multimodal federated learning (FL) aims to enrich model training in FL settings where devices are collecting measurements across multiple modalities (e.g., sensors measuring pressure, motion, and other types of data). However, key challenges to multimodal FL remain unaddressed, particularly in heterogeneous network settings: (i) the set of modalities collected by each device will be diverse, and (ii) communication limitations prevent devices from uploading all their locally trained modality models to the server. In this paper, we propose Federated Multimodal Fusion learning with Selective modality communication (FedMFS), a new multimodal fusion FL methodology that can tackle the above mentioned challenges. The key idea is the introduction of a modality selection criterion for each device, which weighs (i) the impact of the modality, gauged by Shapley value analysis, against (ii) the modality model size as a gauge for communication overhead. This enables FedMFS to flexibly balance performance against communication costs, depending on resource constraints and application requirements. Experiments on the real-world ActionSense dataset demonstrate the ability of FedMFS to achieve comparable accuracy to several baselines while reducing the communication overhead by over 4x. |
| title | FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communication |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing Networking and Internet Architecture |
| url | https://arxiv.org/abs/2310.07048 |