Impact of Leakage on Data Harmonization in Machine Learning Pipelines in Class Imbalance Across Sites
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908373735178240 |
|---|---|
| author | Nieto, Nicolás Eickhoff, Simon B. Jung, Christian Reuter, Martin Diers, Kersten Kelm, Malte Lichtenberg, Artur Raimondo, Federico Patil, Kaustubh R. |
| author_facet | Nieto, Nicolás Eickhoff, Simon B. Jung, Christian Reuter, Martin Diers, Kersten Kelm, Malte Lichtenberg, Artur Raimondo, Federico Patil, Kaustubh R. |
| contents | Machine learning (ML) models benefit from large datasets. Collecting data in biomedical domains is costly and challenging, hence, combining datasets has become a common practice. However, datasets obtained under different conditions could present undesired site-specific variability. Data harmonization methods aim to remove site-specific variance while retaining biologically relevant information. This study evaluates the effectiveness of popularly used ComBat-based methods for harmonizing data in scenarios where the class balance is not equal across sites. We find that these methods struggle with data leakage issues. To overcome this problem, we propose a novel approach PrettYharmonize, designed to harmonize data by pretending the target labels. We validate our approach using controlled datasets designed to benchmark the utility of harmonization. Finally, using real-world MRI and clinical data, we compare leakage-prone methods with PrettYharmonize and show that it achieves comparable performance while avoiding data leakage, particularly in site-target-dependence scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_19643 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Impact of Leakage on Data Harmonization in Machine Learning Pipelines in Class Imbalance Across Sites Nieto, Nicolás Eickhoff, Simon B. Jung, Christian Reuter, Martin Diers, Kersten Kelm, Malte Lichtenberg, Artur Raimondo, Federico Patil, Kaustubh R. Machine Learning Artificial Intelligence Machine learning (ML) models benefit from large datasets. Collecting data in biomedical domains is costly and challenging, hence, combining datasets has become a common practice. However, datasets obtained under different conditions could present undesired site-specific variability. Data harmonization methods aim to remove site-specific variance while retaining biologically relevant information. This study evaluates the effectiveness of popularly used ComBat-based methods for harmonizing data in scenarios where the class balance is not equal across sites. We find that these methods struggle with data leakage issues. To overcome this problem, we propose a novel approach PrettYharmonize, designed to harmonize data by pretending the target labels. We validate our approach using controlled datasets designed to benchmark the utility of harmonization. Finally, using real-world MRI and clinical data, we compare leakage-prone methods with PrettYharmonize and show that it achieves comparable performance while avoiding data leakage, particularly in site-target-dependence scenarios. |
| title | Impact of Leakage on Data Harmonization in Machine Learning Pipelines in Class Imbalance Across Sites |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.19643 |