CleanPatrick: A Benchmark for Image Data Cleaning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913842019172352 |
|---|---|
| author | Gröger, Fabian Lionetti, Simone Gottfrois, Philippe Gonzalez-Jimenez, Alvaro Amruthalingam, Ludovic Goessinger, Elisabeth Victoria Lindemann, Hanna Bargiela, Marie Hofbauer, Marie Badri, Omar Tschandl, Philipp Koochek, Arash Groh, Matthew Navarini, Alexander A. Pouly, Marc |
| author_facet | Gröger, Fabian Lionetti, Simone Gottfrois, Philippe Gonzalez-Jimenez, Alvaro Amruthalingam, Ludovic Goessinger, Elisabeth Victoria Lindemann, Hanna Bargiela, Marie Hofbauer, Marie Badri, Omar Tschandl, Philipp Koochek, Arash Groh, Matthew Navarini, Alexander A. Pouly, Marc |
| contents | Robust machine learning depends on clean data, yet current image data cleaning benchmarks rely on synthetic noise or narrow human studies, limiting comparison and real-world relevance. We introduce CleanPatrick, the first large-scale benchmark for data cleaning in the image domain, built upon the publicly available Fitzpatrick17k dermatology dataset. We collect 496,377 binary annotations from 933 medical crowd workers, identify off-topic samples (4%), near-duplicates (21%), and label errors (22%), and employ an aggregation model inspired by item-response theory followed by expert review to derive high-quality ground truth. CleanPatrick formalizes issue detection as a ranking task and adopts typical ranking metrics mirroring real audit workflows. Benchmarking classical anomaly detectors, perceptual hashing, SSIM, Confident Learning, NoiseRank, and SelfClean, we find that, on CleanPatrick, self-supervised representations excel at near-duplicate detection, classical methods achieve competitive off-topic detection under constrained review budgets, and label-error detection remains an open challenge for fine-grained medical classification. By releasing both the dataset and the evaluation framework, CleanPatrick enables a systematic comparison of image-cleaning strategies and paves the way for more reliable data-centric artificial intelligence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_11034 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CleanPatrick: A Benchmark for Image Data Cleaning Gröger, Fabian Lionetti, Simone Gottfrois, Philippe Gonzalez-Jimenez, Alvaro Amruthalingam, Ludovic Goessinger, Elisabeth Victoria Lindemann, Hanna Bargiela, Marie Hofbauer, Marie Badri, Omar Tschandl, Philipp Koochek, Arash Groh, Matthew Navarini, Alexander A. Pouly, Marc Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Robust machine learning depends on clean data, yet current image data cleaning benchmarks rely on synthetic noise or narrow human studies, limiting comparison and real-world relevance. We introduce CleanPatrick, the first large-scale benchmark for data cleaning in the image domain, built upon the publicly available Fitzpatrick17k dermatology dataset. We collect 496,377 binary annotations from 933 medical crowd workers, identify off-topic samples (4%), near-duplicates (21%), and label errors (22%), and employ an aggregation model inspired by item-response theory followed by expert review to derive high-quality ground truth. CleanPatrick formalizes issue detection as a ranking task and adopts typical ranking metrics mirroring real audit workflows. Benchmarking classical anomaly detectors, perceptual hashing, SSIM, Confident Learning, NoiseRank, and SelfClean, we find that, on CleanPatrick, self-supervised representations excel at near-duplicate detection, classical methods achieve competitive off-topic detection under constrained review budgets, and label-error detection remains an open challenge for fine-grained medical classification. By releasing both the dataset and the evaluation framework, CleanPatrick enables a systematic comparison of image-cleaning strategies and paves the way for more reliable data-centric artificial intelligence. |
| title | CleanPatrick: A Benchmark for Image Data Cleaning |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2505.11034 |