Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.21656 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909370373111808 |
|---|---|
| author | Uematsu, Kazuki Haruki, Kosuke Suzuki, Taiji Kimura, Mitsuhiro Takimoto, Takahiro Nakagawa, Hideyuki |
| author_facet | Uematsu, Kazuki Haruki, Kosuke Suzuki, Taiji Kimura, Mitsuhiro Takimoto, Takahiro Nakagawa, Hideyuki |
| contents | Out-of-distribution (OOD) detection is a critical issue for the stable and reliable operation of systems using a deep neural network (DNN). Although many OOD detection methods have been proposed, it remains unclear how the differences between in-distribution (ID) and OOD samples are generated by each processing step inside DNNs. We experimentally clarify this issue by investigating the layer dependence of feature representations from multiple perspectives. We find that intrinsic low dimensionalization of DNNs is essential for understanding how OOD samples become more distinct from ID samples as features propagate to deeper layers. Based on these observations, we provide a simple picture that consistently explains various properties of OOD samples. Specifically, low-dimensional weights eliminate most information from OOD samples, resulting in misclassifications due to excessive attention to dataset bias. In addition, we demonstrate the utility of dimensionality by proposing a dimensionality-aware OOD detection method based on alignment of features and weights, which consistently achieves high performance for various datasets with lower computational cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_21656 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dimensionality-induced information loss of outliers in deep neural networks Uematsu, Kazuki Haruki, Kosuke Suzuki, Taiji Kimura, Mitsuhiro Takimoto, Takahiro Nakagawa, Hideyuki Machine Learning Out-of-distribution (OOD) detection is a critical issue for the stable and reliable operation of systems using a deep neural network (DNN). Although many OOD detection methods have been proposed, it remains unclear how the differences between in-distribution (ID) and OOD samples are generated by each processing step inside DNNs. We experimentally clarify this issue by investigating the layer dependence of feature representations from multiple perspectives. We find that intrinsic low dimensionalization of DNNs is essential for understanding how OOD samples become more distinct from ID samples as features propagate to deeper layers. Based on these observations, we provide a simple picture that consistently explains various properties of OOD samples. Specifically, low-dimensional weights eliminate most information from OOD samples, resulting in misclassifications due to excessive attention to dataset bias. In addition, we demonstrate the utility of dimensionality by proposing a dimensionality-aware OOD detection method based on alignment of features and weights, which consistently achieves high performance for various datasets with lower computational cost. |
| title | Dimensionality-induced information loss of outliers in deep neural networks |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.21656 |