Mathematical analysis of one-layer neural network with fixed biases, a new activation function and other observations
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910115251093504 |
|---|---|
| author | Macià, Fabricio Nakamura, Shu |
| author_facet | Macià, Fabricio Nakamura, Shu |
| contents | We analyze a simple one-hidden-layer neural network with ReLU activation functions and fixed biases, with one-dimensional input and output. We study both continuous and discrete versions of the model, and we rigorously prove the convergence of the learning process with the $L^2$ squared loss function and the gradient descent procedure. We also prove the spectral bias property for this learning process.
Several conclusions of this analysis are discussed; in particular, regarding the structure and properties that activation functions should possess, as well as the relationships between the spectrum of certain operators and the learning process. Based on this, we also propose an alternative activation function, the full-wave rectified exponential function (FReX), and we discuss the convergence of the gradient descent with this alternative activation function. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_07715 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Mathematical analysis of one-layer neural network with fixed biases, a new activation function and other observations Macià, Fabricio Nakamura, Shu Machine Learning Optimization and Control We analyze a simple one-hidden-layer neural network with ReLU activation functions and fixed biases, with one-dimensional input and output. We study both continuous and discrete versions of the model, and we rigorously prove the convergence of the learning process with the $L^2$ squared loss function and the gradient descent procedure. We also prove the spectral bias property for this learning process. Several conclusions of this analysis are discussed; in particular, regarding the structure and properties that activation functions should possess, as well as the relationships between the spectrum of certain operators and the learning process. Based on this, we also propose an alternative activation function, the full-wave rectified exponential function (FReX), and we discuss the convergence of the gradient descent with this alternative activation function. |
| title | Mathematical analysis of one-layer neural network with fixed biases, a new activation function and other observations |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2604.07715 |