Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.18614 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909123162931200 |
|---|---|
| author | Ali, Hafiz Tiomoko Michieli, Umberto Moon, Ji Joong Kim, Daehyun Ozay, Mete |
| author_facet | Ali, Hafiz Tiomoko Michieli, Umberto Moon, Ji Joong Kim, Daehyun Ozay, Mete |
| contents | The recently discovered Neural collapse (NC) phenomenon states that the last-layer weights of Deep Neural Networks (DNN), converge to the so-called Equiangular Tight Frame (ETF) simplex, at the terminal phase of their training. This ETF geometry is equivalent to vanishing within-class variability of the last layer activations. Inspired by NC properties, we explore in this paper the transferability of DNN models trained with their last layer weight fixed according to ETF. This enforces class separation by eliminating class covariance information, effectively providing implicit regularization. We show that DNN models trained with such a fixed classifier significantly improve transfer performance, particularly on out-of-domain datasets. On a broad range of fine-grained image classification datasets, our approach outperforms i) baseline methods that do not perform any covariance regularization (up to 22%), as well as ii) methods that explicitly whiten covariance of activations throughout training (up to 19%). Our findings suggest that DNNs trained with fixed ETF classifiers offer a powerful mechanism for improving transfer learning across domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_18614 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Deep Neural Network Models Trained With A Fixed Random Classifier Transfer Better Across Domains Ali, Hafiz Tiomoko Michieli, Umberto Moon, Ji Joong Kim, Daehyun Ozay, Mete Machine Learning Computer Vision and Pattern Recognition Neural and Evolutionary Computing The recently discovered Neural collapse (NC) phenomenon states that the last-layer weights of Deep Neural Networks (DNN), converge to the so-called Equiangular Tight Frame (ETF) simplex, at the terminal phase of their training. This ETF geometry is equivalent to vanishing within-class variability of the last layer activations. Inspired by NC properties, we explore in this paper the transferability of DNN models trained with their last layer weight fixed according to ETF. This enforces class separation by eliminating class covariance information, effectively providing implicit regularization. We show that DNN models trained with such a fixed classifier significantly improve transfer performance, particularly on out-of-domain datasets. On a broad range of fine-grained image classification datasets, our approach outperforms i) baseline methods that do not perform any covariance regularization (up to 22%), as well as ii) methods that explicitly whiten covariance of activations throughout training (up to 19%). Our findings suggest that DNNs trained with fixed ETF classifiers offer a powerful mechanism for improving transfer learning across domains. |
| title | Deep Neural Network Models Trained With A Fixed Random Classifier Transfer Better Across Domains |
| topic | Machine Learning Computer Vision and Pattern Recognition Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2402.18614 |