Embeddings between Barron spaces with higher order activation functions
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866907947586551808 |
|---|---|
| author | Heeringa, Tjeerd Jan Spek, Len Schwenninger, Felix Brune, Christoph |
| author_facet | Heeringa, Tjeerd Jan Spek, Len Schwenninger, Felix Brune, Christoph |
| contents | The approximation properties of infinitely wide shallow neural networks heavily depend on the choice of the activation function. To understand this influence, we study embeddings between Barron spaces with different activation functions. These embeddings are proven by providing push-forward maps on the measures $μ$ used to represent functions $f$. An activation function of particular interest is the rectified power unit ($\operatorname{RePU}$) given by $\operatorname{RePU}_s(x)=\max(0,x)^s$. For many commonly used activation functions, the well-known Taylor remainder theorem can be used to construct a push-forward map, which allows us to prove the embedding of the associated Barron space into a Barron space with a $\operatorname{RePU}$ as activation function. Moreover, the Barron spaces associated with the $\operatorname{RePU}_s$ have a hierarchical structure similar to the Sobolev spaces $H^m$. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_15839 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Embeddings between Barron spaces with higher order activation functions Heeringa, Tjeerd Jan Spek, Len Schwenninger, Felix Brune, Christoph Machine Learning Functional Analysis 46E35 (Primary) 46E15, 46G12 (Secondary) I.2.6; G.1.9 The approximation properties of infinitely wide shallow neural networks heavily depend on the choice of the activation function. To understand this influence, we study embeddings between Barron spaces with different activation functions. These embeddings are proven by providing push-forward maps on the measures $μ$ used to represent functions $f$. An activation function of particular interest is the rectified power unit ($\operatorname{RePU}$) given by $\operatorname{RePU}_s(x)=\max(0,x)^s$. For many commonly used activation functions, the well-known Taylor remainder theorem can be used to construct a push-forward map, which allows us to prove the embedding of the associated Barron space into a Barron space with a $\operatorname{RePU}$ as activation function. Moreover, the Barron spaces associated with the $\operatorname{RePU}_s$ have a hierarchical structure similar to the Sobolev spaces $H^m$. |
| title | Embeddings between Barron spaces with higher order activation functions |
| topic | Machine Learning Functional Analysis 46E35 (Primary) 46E15, 46G12 (Secondary) I.2.6; G.1.9 |
| url | https://arxiv.org/abs/2305.15839 |