A ZeNN architecture to avoid the Gaussian trap
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908473728434176 |
|---|---|
| author | Carvalho, Luís Costa, João L. Mourão, José Oliveira, Gonçalo |
| author_facet | Carvalho, Luís Costa, João L. Mourão, José Oliveira, Gonçalo |
| contents | We propose a new simple architecture, Zeta Neural Networks (ZeNNs), in order to overcome several shortcomings of standard multi-layer perceptrons (MLPs). Namely, in the large width limit, MLPs are non-parametric, they do not have a well-defined pointwise limit, they lose non-Gaussian attributes and become unable to perform feature learning; moreover, finite width MLPs perform poorly in learning high frequencies. The new ZeNN architecture is inspired by three simple principles from harmonic analysis:
i) Enumerate the perceptons and introduce a non-learnable weight to enforce convergence;
ii) Introduce a scaling (or frequency) factor;
iii) Choose activation functions that lead to near orthogonal systems.
We will show that these ideas allow us to fix the referred shortcomings of MLPs. In fact, in the infinite width limit, ZeNNs converge pointwise, they exhibit a rich asymptotic structure beyond Gaussianity, and perform feature learning. Moreover, when appropriate activation functions are chosen, (finite width) ZeNNs excel at learning high-frequency features of functions with low dimensional domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_20553 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A ZeNN architecture to avoid the Gaussian trap Carvalho, Luís Costa, João L. Mourão, José Oliveira, Gonçalo Machine Learning Probability 68T07, 68T01 I.2.0; G.0 We propose a new simple architecture, Zeta Neural Networks (ZeNNs), in order to overcome several shortcomings of standard multi-layer perceptrons (MLPs). Namely, in the large width limit, MLPs are non-parametric, they do not have a well-defined pointwise limit, they lose non-Gaussian attributes and become unable to perform feature learning; moreover, finite width MLPs perform poorly in learning high frequencies. The new ZeNN architecture is inspired by three simple principles from harmonic analysis: i) Enumerate the perceptons and introduce a non-learnable weight to enforce convergence; ii) Introduce a scaling (or frequency) factor; iii) Choose activation functions that lead to near orthogonal systems. We will show that these ideas allow us to fix the referred shortcomings of MLPs. In fact, in the infinite width limit, ZeNNs converge pointwise, they exhibit a rich asymptotic structure beyond Gaussianity, and perform feature learning. Moreover, when appropriate activation functions are chosen, (finite width) ZeNNs excel at learning high-frequency features of functions with low dimensional domains. |
| title | A ZeNN architecture to avoid the Gaussian trap |
| topic | Machine Learning Probability 68T07, 68T01 I.2.0; G.0 |
| url | https://arxiv.org/abs/2505.20553 |