Retraining with Predicted Hard Labels Provably Increases Model Accuracy
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913825849081856 |
|---|---|
| author | Das, Rudrajit Dhillon, Inderjit S. Epasto, Alessandro Javanmard, Adel Mao, Jieming Mirrokni, Vahab Sanghavi, Sujay Zhong, Peilin |
| author_facet | Das, Rudrajit Dhillon, Inderjit S. Epasto, Alessandro Javanmard, Adel Mao, Jieming Mirrokni, Vahab Sanghavi, Sujay Zhong, Peilin |
| contents | The performance of a model trained with noisy labels is often improved by simply \textit{retraining} the model with its \textit{own predicted hard labels} (i.e., 1/0 labels). Yet, a detailed theoretical characterization of this phenomenon is lacking. In this paper, we theoretically analyze retraining in a linearly separable binary classification setting with randomly corrupted labels given to us and prove that retraining can improve the population accuracy obtained by initially training with the given (noisy) labels. To the best of our knowledge, this is the first such theoretical result. Retraining finds application in improving training with local label differential privacy (DP) which involves training with noisy labels. We empirically show that retraining selectively on the samples for which the predicted label matches the given label significantly improves label DP training at no extra privacy cost; we call this consensus-based retraining. As an example, when training ResNet-18 on CIFAR-100 with $ε=3$ label DP, we obtain more than 6% improvement in accuracy with consensus-based retraining. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_11206 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Retraining with Predicted Hard Labels Provably Increases Model Accuracy Das, Rudrajit Dhillon, Inderjit S. Epasto, Alessandro Javanmard, Adel Mao, Jieming Mirrokni, Vahab Sanghavi, Sujay Zhong, Peilin Machine Learning Cryptography and Security The performance of a model trained with noisy labels is often improved by simply \textit{retraining} the model with its \textit{own predicted hard labels} (i.e., 1/0 labels). Yet, a detailed theoretical characterization of this phenomenon is lacking. In this paper, we theoretically analyze retraining in a linearly separable binary classification setting with randomly corrupted labels given to us and prove that retraining can improve the population accuracy obtained by initially training with the given (noisy) labels. To the best of our knowledge, this is the first such theoretical result. Retraining finds application in improving training with local label differential privacy (DP) which involves training with noisy labels. We empirically show that retraining selectively on the samples for which the predicted label matches the given label significantly improves label DP training at no extra privacy cost; we call this consensus-based retraining. As an example, when training ResNet-18 on CIFAR-100 with $ε=3$ label DP, we obtain more than 6% improvement in accuracy with consensus-based retraining. |
| title | Retraining with Predicted Hard Labels Provably Increases Model Accuracy |
| topic | Machine Learning Cryptography and Security |
| url | https://arxiv.org/abs/2406.11206 |