Don't Fear Peculiar Activation Functions: EUAF and Beyond
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916322223325184 |
|---|---|
| author | Wang, Qianchao Zhang, Shijun Zeng, Dong Xie, Zhaoheng Guo, Hengtao Fan, Feng-Lei Zeng, Tieyong |
| author_facet | Wang, Qianchao Zhang, Shijun Zeng, Dong Xie, Zhaoheng Guo, Hengtao Fan, Feng-Lei Zeng, Tieyong |
| contents | In this paper, we propose a new super-expressive activation function called the Parametric Elementary Universal Activation Function (PEUAF). We demonstrate the effectiveness of PEUAF through systematic and comprehensive experiments on various industrial and image datasets, including CIFAR10, Tiny-ImageNet, and ImageNet. Moreover, we significantly generalize the family of super-expressive activation functions, whose existence has been demonstrated in several recent works by showing that any continuous function can be approximated to any desired accuracy by a fixed-size network with a specific super-expressive activation function. Specifically, our work addresses two major bottlenecks in impeding the development of super-expressive activation functions: the limited identification of super-expressive functions, which raises doubts about their broad applicability, and their often peculiar forms, which lead to skepticism regarding their scalability and practicality in real-world applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_09580 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Don't Fear Peculiar Activation Functions: EUAF and Beyond Wang, Qianchao Zhang, Shijun Zeng, Dong Xie, Zhaoheng Guo, Hengtao Fan, Feng-Lei Zeng, Tieyong Computer Vision and Pattern Recognition Artificial Intelligence In this paper, we propose a new super-expressive activation function called the Parametric Elementary Universal Activation Function (PEUAF). We demonstrate the effectiveness of PEUAF through systematic and comprehensive experiments on various industrial and image datasets, including CIFAR10, Tiny-ImageNet, and ImageNet. Moreover, we significantly generalize the family of super-expressive activation functions, whose existence has been demonstrated in several recent works by showing that any continuous function can be approximated to any desired accuracy by a fixed-size network with a specific super-expressive activation function. Specifically, our work addresses two major bottlenecks in impeding the development of super-expressive activation functions: the limited identification of super-expressive functions, which raises doubts about their broad applicability, and their often peculiar forms, which lead to skepticism regarding their scalability and practicality in real-world applications. |
| title | Don't Fear Peculiar Activation Functions: EUAF and Beyond |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2407.09580 |