FlexAct: Why Learn when you can Pick?
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908756949860352 |
|---|---|
| author | Kumar, Ramnath Ritscher, Kyle Judy, Junmin Liu, Lawrence Hsieh, Cho-Jui |
| author_facet | Kumar, Ramnath Ritscher, Kyle Judy, Junmin Liu, Lawrence Hsieh, Cho-Jui |
| contents | Learning activation functions has emerged as a promising direction in deep learning, allowing networks to adapt activation mechanisms to task-specific demands. In this work, we introduce a novel framework that employs the Gumbel-Softmax trick to enable discrete yet differentiable selection among a predefined set of activation functions during training. Our method dynamically learns the optimal activation function independently of the input, thereby enhancing both predictive accuracy and architectural flexibility. Experiments on synthetic datasets show that our model consistently selects the most suitable activation function, underscoring its effectiveness. These results connect theoretical advances with practical utility, paving the way for more adaptive and modular neural architectures in complex learning scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_06441 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | FlexAct: Why Learn when you can Pick? Kumar, Ramnath Ritscher, Kyle Judy, Junmin Liu, Lawrence Hsieh, Cho-Jui Machine Learning Learning activation functions has emerged as a promising direction in deep learning, allowing networks to adapt activation mechanisms to task-specific demands. In this work, we introduce a novel framework that employs the Gumbel-Softmax trick to enable discrete yet differentiable selection among a predefined set of activation functions during training. Our method dynamically learns the optimal activation function independently of the input, thereby enhancing both predictive accuracy and architectural flexibility. Experiments on synthetic datasets show that our model consistently selects the most suitable activation function, underscoring its effectiveness. These results connect theoretical advances with practical utility, paving the way for more adaptive and modular neural architectures in complex learning scenarios. |
| title | FlexAct: Why Learn when you can Pick? |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2601.06441 |