FedMedICL: Towards Holistic Evaluation of Distribution Shifts in Federated Medical Imaging
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910523843411968 |
|---|---|
| author | Alhamoud, Kumail Ghunaim, Yasir Alfarra, Motasem Hartvigsen, Thomas Torr, Philip Ghanem, Bernard Bibi, Adel Ghassemi, Marzyeh |
| author_facet | Alhamoud, Kumail Ghunaim, Yasir Alfarra, Motasem Hartvigsen, Thomas Torr, Philip Ghanem, Bernard Bibi, Adel Ghassemi, Marzyeh |
| contents | For medical imaging AI models to be clinically impactful, they must generalize. However, this goal is hindered by (i) diverse types of distribution shifts, such as temporal, demographic, and label shifts, and (ii) limited diversity in datasets that are siloed within single medical institutions. While these limitations have spurred interest in federated learning, current evaluation benchmarks fail to evaluate different shifts simultaneously. However, in real healthcare settings, multiple types of shifts co-exist, yet their impact on medical imaging performance remains unstudied. In response, we introduce FedMedICL, a unified framework and benchmark to holistically evaluate federated medical imaging challenges, simultaneously capturing label, demographic, and temporal distribution shifts. We comprehensively evaluate several popular methods on six diverse medical imaging datasets (totaling 550 GPU hours). Furthermore, we use FedMedICL to simulate COVID-19 propagation across hospitals and evaluate whether methods can adapt to pandemic changes in disease prevalence. We find that a simple batch balancing technique surpasses advanced methods in average performance across FedMedICL experiments. This finding questions the applicability of results from previous, narrow benchmarks in real-world medical settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_08822 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FedMedICL: Towards Holistic Evaluation of Distribution Shifts in Federated Medical Imaging Alhamoud, Kumail Ghunaim, Yasir Alfarra, Motasem Hartvigsen, Thomas Torr, Philip Ghanem, Bernard Bibi, Adel Ghassemi, Marzyeh Image and Video Processing Artificial Intelligence Computer Vision and Pattern Recognition For medical imaging AI models to be clinically impactful, they must generalize. However, this goal is hindered by (i) diverse types of distribution shifts, such as temporal, demographic, and label shifts, and (ii) limited diversity in datasets that are siloed within single medical institutions. While these limitations have spurred interest in federated learning, current evaluation benchmarks fail to evaluate different shifts simultaneously. However, in real healthcare settings, multiple types of shifts co-exist, yet their impact on medical imaging performance remains unstudied. In response, we introduce FedMedICL, a unified framework and benchmark to holistically evaluate federated medical imaging challenges, simultaneously capturing label, demographic, and temporal distribution shifts. We comprehensively evaluate several popular methods on six diverse medical imaging datasets (totaling 550 GPU hours). Furthermore, we use FedMedICL to simulate COVID-19 propagation across hospitals and evaluate whether methods can adapt to pandemic changes in disease prevalence. We find that a simple batch balancing technique surpasses advanced methods in average performance across FedMedICL experiments. This finding questions the applicability of results from previous, narrow benchmarks in real-world medical settings. |
| title | FedMedICL: Towards Holistic Evaluation of Distribution Shifts in Federated Medical Imaging |
| topic | Image and Video Processing Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.08822 |