dopanim: A Dataset of Doppelganger Animals with Noisy Annotations from Multiple Humans
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913452812926976 |
|---|---|
| author | Herde, Marek Huseljic, Denis Rauch, Lukas Sick, Bernhard |
| author_facet | Herde, Marek Huseljic, Denis Rauch, Lukas Sick, Bernhard |
| contents | Human annotators typically provide annotated data for training machine learning models, such as neural networks. Yet, human annotations are subject to noise, impairing generalization performances. Methodological research on approaches counteracting noisy annotations requires corresponding datasets for a meaningful empirical evaluation. Consequently, we introduce a novel benchmark dataset, dopanim, consisting of about 15,750 animal images of 15 classes with ground truth labels. For approximately 10,500 of these images, 20 humans provided over 52,000 annotations with an accuracy of circa 67%. Its key attributes include (1) the challenging task of classifying doppelganger animals, (2) human-estimated likelihoods as annotations, and (3) annotator metadata. We benchmark well-known multi-annotator learning approaches using seven variants of this dataset and outline further evaluation use cases such as learning beyond hard class labels and active learning. Our dataset and a comprehensive codebase are publicly available to emulate the data collection process and to reproduce all empirical results. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_20950 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | dopanim: A Dataset of Doppelganger Animals with Noisy Annotations from Multiple Humans Herde, Marek Huseljic, Denis Rauch, Lukas Sick, Bernhard Computer Vision and Pattern Recognition Human annotators typically provide annotated data for training machine learning models, such as neural networks. Yet, human annotations are subject to noise, impairing generalization performances. Methodological research on approaches counteracting noisy annotations requires corresponding datasets for a meaningful empirical evaluation. Consequently, we introduce a novel benchmark dataset, dopanim, consisting of about 15,750 animal images of 15 classes with ground truth labels. For approximately 10,500 of these images, 20 humans provided over 52,000 annotations with an accuracy of circa 67%. Its key attributes include (1) the challenging task of classifying doppelganger animals, (2) human-estimated likelihoods as annotations, and (3) annotator metadata. We benchmark well-known multi-annotator learning approaches using seven variants of this dataset and outline further evaluation use cases such as learning beyond hard class labels and active learning. Our dataset and a comprehensive codebase are publicly available to emulate the data collection process and to reproduce all empirical results. |
| title | dopanim: A Dataset of Doppelganger Animals with Noisy Annotations from Multiple Humans |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.20950 |