Using Early Readouts to Mediate Featural Bias in Distillation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916523830935552 |
|---|---|
| author | Tiwari, Rishabh Sivasubramanian, Durga Mekala, Anmol Ramakrishnan, Ganesh Shenoy, Pradeep |
| author_facet | Tiwari, Rishabh Sivasubramanian, Durga Mekala, Anmol Ramakrishnan, Ganesh Shenoy, Pradeep |
| contents | Deep networks tend to learn spurious feature-label correlations in real-world supervised learning tasks. This vulnerability is aggravated in distillation, where a student model may have lesser representational capacity than the corresponding teacher model. Often, knowledge of specific spurious correlations is used to reweight instances & rebalance the learning process. We propose a novel early readout mechanism whereby we attempt to predict the label using representations from earlier network layers. We show that these early readouts automatically identify problem instances or groups in the form of confident, incorrect predictions. Leveraging these signals to modulate the distillation loss on an instance level allows us to substantially improve not only group fairness measures across benchmark datasets, but also overall accuracy of the student model. We also provide secondary analyses that bring insight into the role of feature learning in supervision and distillation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_18590 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Using Early Readouts to Mediate Featural Bias in Distillation Tiwari, Rishabh Sivasubramanian, Durga Mekala, Anmol Ramakrishnan, Ganesh Shenoy, Pradeep Machine Learning Artificial Intelligence Deep networks tend to learn spurious feature-label correlations in real-world supervised learning tasks. This vulnerability is aggravated in distillation, where a student model may have lesser representational capacity than the corresponding teacher model. Often, knowledge of specific spurious correlations is used to reweight instances & rebalance the learning process. We propose a novel early readout mechanism whereby we attempt to predict the label using representations from earlier network layers. We show that these early readouts automatically identify problem instances or groups in the form of confident, incorrect predictions. Leveraging these signals to modulate the distillation loss on an instance level allows us to substantially improve not only group fairness measures across benchmark datasets, but also overall accuracy of the student model. We also provide secondary analyses that bring insight into the role of feature learning in supervision and distillation. |
| title | Using Early Readouts to Mediate Featural Bias in Distillation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2310.18590 |