Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selection
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929439290425344 |
|---|---|
| author | Fan, Yunfeng Xu, Wenchao Wang, Haozhao Huo, Fushuo Chen, Jinyu Guo, Song |
| author_facet | Fan, Yunfeng Xu, Wenchao Wang, Haozhao Huo, Fushuo Chen, Jinyu Guo, Song |
| contents | Selecting proper clients to participate in each federated learning (FL) round is critical to effectively harness a broad range of distributed data. Existing client selection methods simply consider the mining of distributed uni-modal data, yet, their effectiveness may diminish in multi-modal FL (MFL) as the modality imbalance problem not only impedes the collaborative local training but also leads to a severe global modality-level bias. We empirically reveal that local training with a certain single modality may contribute more to the global model than training with all local modalities. To effectively exploit the distributed multiple modalities, we propose a novel Balanced Modality Selection framework for MFL (BMSFed) to overcome the modal bias. On the one hand, we introduce a modal enhancement loss during local training to alleviate local imbalance based on the aggregated global prototypes. On the other hand, we propose the modality selection aiming to select subsets of local modalities with great diversity and achieving global modal balance simultaneously. Our extensive experiments on audio-visual, colored-gray, and front-back datasets showcase the superiority of BMSFed over baselines and its effectiveness in multi-modal data exploitation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_00403 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selection Fan, Yunfeng Xu, Wenchao Wang, Haozhao Huo, Fushuo Chen, Jinyu Guo, Song Machine Learning Computer Vision and Pattern Recognition Multimedia Selecting proper clients to participate in each federated learning (FL) round is critical to effectively harness a broad range of distributed data. Existing client selection methods simply consider the mining of distributed uni-modal data, yet, their effectiveness may diminish in multi-modal FL (MFL) as the modality imbalance problem not only impedes the collaborative local training but also leads to a severe global modality-level bias. We empirically reveal that local training with a certain single modality may contribute more to the global model than training with all local modalities. To effectively exploit the distributed multiple modalities, we propose a novel Balanced Modality Selection framework for MFL (BMSFed) to overcome the modal bias. On the one hand, we introduce a modal enhancement loss during local training to alleviate local imbalance based on the aggregated global prototypes. On the other hand, we propose the modality selection aiming to select subsets of local modalities with great diversity and achieving global modal balance simultaneously. Our extensive experiments on audio-visual, colored-gray, and front-back datasets showcase the superiority of BMSFed over baselines and its effectiveness in multi-modal data exploitation. |
| title | Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selection |
| topic | Machine Learning Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2401.00403 |