GmNet: Revisiting Gating Mechanisms From A Frequency View
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908854330064896 |
|---|---|
| author | Wang, Yifan Ma, Xu Zhang, Yitian Wang, Zhongruo Kim, Sung-Cheol Mirjalili, Vahid Renganathan, Vidya Fu, Yun |
| author_facet | Wang, Yifan Ma, Xu Zhang, Yitian Wang, Zhongruo Kim, Sung-Cheol Mirjalili, Vahid Renganathan, Vidya Fu, Yun |
| contents | Gating mechanisms have emerged as an effective strategy integrated into model designs beyond recurrent neural networks for addressing long-range dependency problems. In a broad understanding, it provides adaptive control over the information flow while maintaining computational efficiency. However, there is a lack of theoretical analysis on how the gating mechanism works in neural networks. In this paper, inspired by the \textit{convolution theorem}, we systematically explore the effect of gating mechanisms on the training dynamics of neural networks from a frequency perspective. We investigate the interact between the element-wise product and activation functions in managing the responses to different frequency components. Leveraging these insights, we propose a Gating Mechanism Network (GmNet), a lightweight model designed to efficiently utilize the information of various frequency components. It minimizes the low-frequency bias present in existing lightweight models. GmNet achieves impressive performance in terms of both effectiveness and efficiency in the image classification task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_22841 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GmNet: Revisiting Gating Mechanisms From A Frequency View Wang, Yifan Ma, Xu Zhang, Yitian Wang, Zhongruo Kim, Sung-Cheol Mirjalili, Vahid Renganathan, Vidya Fu, Yun Computer Vision and Pattern Recognition Gating mechanisms have emerged as an effective strategy integrated into model designs beyond recurrent neural networks for addressing long-range dependency problems. In a broad understanding, it provides adaptive control over the information flow while maintaining computational efficiency. However, there is a lack of theoretical analysis on how the gating mechanism works in neural networks. In this paper, inspired by the \textit{convolution theorem}, we systematically explore the effect of gating mechanisms on the training dynamics of neural networks from a frequency perspective. We investigate the interact between the element-wise product and activation functions in managing the responses to different frequency components. Leveraging these insights, we propose a Gating Mechanism Network (GmNet), a lightweight model designed to efficiently utilize the information of various frequency components. It minimizes the low-frequency bias present in existing lightweight models. GmNet achieves impressive performance in terms of both effectiveness and efficiency in the image classification task. |
| title | GmNet: Revisiting Gating Mechanisms From A Frequency View |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.22841 |