Understanding Two-Layer Neural Networks with Smooth Activation Functions
Fuente:
arXiv
Guardado en:
| Autor principal: | |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913948806152192 |
|---|---|
| author | Huang, Changcun |
| author_facet | Huang, Changcun |
| contents | This paper aims to understand the training solution, which is obtained by the back-propagation algorithm, of two-layer neural networks whose hidden layer is composed of the units with smooth activation functions, including the usual sigmoid type most commonly used before the advent of ReLUs. The mechanism contains four main principles: construction of Taylor series expansions, strict partial order of knots, smooth-spline implementation and smooth-continuity restriction. The universal approximation for arbitrary input dimensionality is proved and experimental verification is given, through which the mystery of ``black box'' of the solution space is largely revealed. The new proofs employed also enrich approximation theory. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_14177 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Understanding Two-Layer Neural Networks with Smooth Activation Functions Huang, Changcun Machine Learning Artificial Intelligence Numerical Analysis 68T07(Primary), 41A15(Secondary) I.2.6; G.1.2 This paper aims to understand the training solution, which is obtained by the back-propagation algorithm, of two-layer neural networks whose hidden layer is composed of the units with smooth activation functions, including the usual sigmoid type most commonly used before the advent of ReLUs. The mechanism contains four main principles: construction of Taylor series expansions, strict partial order of knots, smooth-spline implementation and smooth-continuity restriction. The universal approximation for arbitrary input dimensionality is proved and experimental verification is given, through which the mystery of ``black box'' of the solution space is largely revealed. The new proofs employed also enrich approximation theory. |
| title | Understanding Two-Layer Neural Networks with Smooth Activation Functions |
| topic | Machine Learning Artificial Intelligence Numerical Analysis 68T07(Primary), 41A15(Secondary) I.2.6; G.1.2 |
| url | https://arxiv.org/abs/2507.14177 |