LayerAct: Advanced Activation Mechanism for Robust Inference of CNNs
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909500733128704 |
|---|---|
| author | Yoon, Kihyuk Lim, Chiehyeon |
| author_facet | Yoon, Kihyuk Lim, Chiehyeon |
| contents | In this work, we propose a novel activation mechanism called LayerAct for CNNs. This approach is motivated by our theoretical and experimental analyses, which demonstrate that Layer Normalization (LN) can mitigate a limitation of existing activation functions regarding noise robustness. However, LN is known to be disadvantageous in CNNs due to its tendency to make activation outputs homogeneous. The proposed method is designed to be more robust than existing activation functions by reducing the upper bound of influence caused by input shifts without inheriting LN's limitation. We provide analyses and experiments showing that LayerAct functions exhibit superior robustness compared to ElementAct functions. Experimental results on three clean and noisy benchmark datasets for image classification tasks indicate that LayerAct functions outperform other activation functions in handling noisy datasets while achieving superior performance on clean datasets in most cases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_04940 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | LayerAct: Advanced Activation Mechanism for Robust Inference of CNNs Yoon, Kihyuk Lim, Chiehyeon Machine Learning Computer Vision and Pattern Recognition Neural and Evolutionary Computing 68T07 (Primary) 68T45 (Secondary) In this work, we propose a novel activation mechanism called LayerAct for CNNs. This approach is motivated by our theoretical and experimental analyses, which demonstrate that Layer Normalization (LN) can mitigate a limitation of existing activation functions regarding noise robustness. However, LN is known to be disadvantageous in CNNs due to its tendency to make activation outputs homogeneous. The proposed method is designed to be more robust than existing activation functions by reducing the upper bound of influence caused by input shifts without inheriting LN's limitation. We provide analyses and experiments showing that LayerAct functions exhibit superior robustness compared to ElementAct functions. Experimental results on three clean and noisy benchmark datasets for image classification tasks indicate that LayerAct functions outperform other activation functions in handling noisy datasets while achieving superior performance on clean datasets in most cases. |
| title | LayerAct: Advanced Activation Mechanism for Robust Inference of CNNs |
| topic | Machine Learning Computer Vision and Pattern Recognition Neural and Evolutionary Computing 68T07 (Primary) 68T45 (Secondary) |
| url | https://arxiv.org/abs/2306.04940 |