Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.13140 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910749931077632 |
|---|---|
| author | Lim, Gordon Larson, Stefan Leach, Kevin |
| author_facet | Lim, Gordon Larson, Stefan Leach, Kevin |
| contents | Tobacco3482 is a widely used document classification benchmark dataset. However, our manual inspection of the entire dataset uncovers widespread ontological issues, especially large amounts of annotation label problems in the dataset. We establish data label guidelines and find that 11.7% of the dataset is improperly annotated and should either have an unknown label or a corrected label, and 16.7% of samples in the dataset have multiple valid labels. We then analyze the mistakes of a top-performing model and find that 35% of the model's mistakes can be directly attributed to these label issues, highlighting the inherent problems with using a noisily labeled dataset as a benchmark. Supplementary material, including dataset annotations and code, is available at https://github.com/gordon-lim/tobacco3482-mistakes/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13140 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Label Errors in the Tobacco3482 Dataset Lim, Gordon Larson, Stefan Leach, Kevin Computer Vision and Pattern Recognition Tobacco3482 is a widely used document classification benchmark dataset. However, our manual inspection of the entire dataset uncovers widespread ontological issues, especially large amounts of annotation label problems in the dataset. We establish data label guidelines and find that 11.7% of the dataset is improperly annotated and should either have an unknown label or a corrected label, and 16.7% of samples in the dataset have multiple valid labels. We then analyze the mistakes of a top-performing model and find that 35% of the model's mistakes can be directly attributed to these label issues, highlighting the inherent problems with using a noisily labeled dataset as a benchmark. Supplementary material, including dataset annotations and code, is available at https://github.com/gordon-lim/tobacco3482-mistakes/. |
| title | Label Errors in the Tobacco3482 Dataset |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.13140 |