Optimizing Performance of Feedforward and Convolutional Neural Networks through Dynamic Activation Functions
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917592444174336 |
|---|---|
| author | Rane, Chinmay Tyagi, Kanishka Manry, Michael |
| author_facet | Rane, Chinmay Tyagi, Kanishka Manry, Michael |
| contents | Deep learning training training algorithms are a huge success in recent years in many fields including speech, text,image video etc. Deeper and deeper layers are proposed with huge success with resnet structures having around 152 layers. Shallow convolution neural networks(CNN's) are still an active research, where some phenomena are still unexplained. Activation functions used in the network are of utmost importance, as they provide non linearity to the networks. Relu's are the most commonly used activation function.We show a complex piece-wise linear(PWL) activation in the hidden layer. We show that these PWL activations work much better than relu activations in our networks for convolution neural networks and multilayer perceptrons. Result comparison in PyTorch for shallow and deep CNNs are given to further strengthen our case. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2308_05724 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Optimizing Performance of Feedforward and Convolutional Neural Networks through Dynamic Activation Functions Rane, Chinmay Tyagi, Kanishka Manry, Michael Machine Learning Computational Engineering, Finance, and Science Neural and Evolutionary Computing Deep learning training training algorithms are a huge success in recent years in many fields including speech, text,image video etc. Deeper and deeper layers are proposed with huge success with resnet structures having around 152 layers. Shallow convolution neural networks(CNN's) are still an active research, where some phenomena are still unexplained. Activation functions used in the network are of utmost importance, as they provide non linearity to the networks. Relu's are the most commonly used activation function.We show a complex piece-wise linear(PWL) activation in the hidden layer. We show that these PWL activations work much better than relu activations in our networks for convolution neural networks and multilayer perceptrons. Result comparison in PyTorch for shallow and deep CNNs are given to further strengthen our case. |
| title | Optimizing Performance of Feedforward and Convolutional Neural Networks through Dynamic Activation Functions |
| topic | Machine Learning Computational Engineering, Finance, and Science Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2308.05724 |