Activations Through Extensions: A Framework To Boost Performance Of Neural Networks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916359488667648 |
|---|---|
| author | Kamanchi, Chandramouli Mukherjee, Sumanta Sampath, Kameshwaran Dayama, Pankaj Jati, Arindam Ekambaram, Vijay Phan, Dzung |
| author_facet | Kamanchi, Chandramouli Mukherjee, Sumanta Sampath, Kameshwaran Dayama, Pankaj Jati, Arindam Ekambaram, Vijay Phan, Dzung |
| contents | Activation functions are non-linearities in neural networks that allow them to learn complex mapping between inputs and outputs. Typical choices for activation functions are ReLU, Tanh, Sigmoid etc., where the choice generally depends on the application domain. In this work, we propose a framework/strategy that unifies several works on activation functions and theoretically explains the performance benefits of these works. We also propose novel techniques that originate from the framework and allow us to obtain ``extensions'' (i.e. special generalizations of a given neural network) of neural networks through operations on activation functions. We theoretically and empirically show that ``extensions'' of neural networks have performance benefits compared to vanilla neural networks with insignificant space and time complexity costs on standard test functions. We also show the benefits of neural network ``extensions'' in the time-series domain on real-world datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_03599 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Activations Through Extensions: A Framework To Boost Performance Of Neural Networks Kamanchi, Chandramouli Mukherjee, Sumanta Sampath, Kameshwaran Dayama, Pankaj Jati, Arindam Ekambaram, Vijay Phan, Dzung Machine Learning Artificial Intelligence Numerical Analysis Neural and Evolutionary Computing Activation functions are non-linearities in neural networks that allow them to learn complex mapping between inputs and outputs. Typical choices for activation functions are ReLU, Tanh, Sigmoid etc., where the choice generally depends on the application domain. In this work, we propose a framework/strategy that unifies several works on activation functions and theoretically explains the performance benefits of these works. We also propose novel techniques that originate from the framework and allow us to obtain ``extensions'' (i.e. special generalizations of a given neural network) of neural networks through operations on activation functions. We theoretically and empirically show that ``extensions'' of neural networks have performance benefits compared to vanilla neural networks with insignificant space and time complexity costs on standard test functions. We also show the benefits of neural network ``extensions'' in the time-series domain on real-world datasets. |
| title | Activations Through Extensions: A Framework To Boost Performance Of Neural Networks |
| topic | Machine Learning Artificial Intelligence Numerical Analysis Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2408.03599 |