A Lipschitz spaces view of infinitely wide shallow neural networks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917147846901760 |
|---|---|
| author | Bartolucci, Francesca Carioni, Marcello Iglesias, José A. Korolev, Yury Naldi, Emanuele Vigogna, Stefano |
| author_facet | Bartolucci, Francesca Carioni, Marcello Iglesias, José A. Korolev, Yury Naldi, Emanuele Vigogna, Stefano |
| contents | We revisit the mean field parametrization of shallow neural networks, using signed measures on unbounded parameter spaces and duality pairings that take into account the regularity and growth of activation functions. This setting directly leads to the use of unbalanced Kantorovich-Rubinstein norms defined by duality with Lipschitz functions, and of spaces of measures dual to those of continuous functions with controlled growth. These allow to make transparent the need for total variation and moment bounds or penalization to obtain existence of minimizers of variational formulations, under which we prove a compactness result in strong Kantorovich-Rubinstein norm, and in the absence of which we show several examples demonstrating undesirable behavior. Further, the Kantorovich-Rubinstein setting enables us to combine the advantages of a completely linear parametrization and ensuing reproducing kernel Banach space framework with optimal transport insights. We showcase this synergy with representer theorems and uniform large data limits for empirical risk minimization, and in proposed formulations for distillation and fusion applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_14591 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Lipschitz spaces view of infinitely wide shallow neural networks Bartolucci, Francesca Carioni, Marcello Iglesias, José A. Korolev, Yury Naldi, Emanuele Vigogna, Stefano Functional Analysis Machine Learning 68T07, 46E27, 46B20 We revisit the mean field parametrization of shallow neural networks, using signed measures on unbounded parameter spaces and duality pairings that take into account the regularity and growth of activation functions. This setting directly leads to the use of unbalanced Kantorovich-Rubinstein norms defined by duality with Lipschitz functions, and of spaces of measures dual to those of continuous functions with controlled growth. These allow to make transparent the need for total variation and moment bounds or penalization to obtain existence of minimizers of variational formulations, under which we prove a compactness result in strong Kantorovich-Rubinstein norm, and in the absence of which we show several examples demonstrating undesirable behavior. Further, the Kantorovich-Rubinstein setting enables us to combine the advantages of a completely linear parametrization and ensuing reproducing kernel Banach space framework with optimal transport insights. We showcase this synergy with representer theorems and uniform large data limits for empirical risk minimization, and in proposed formulations for distillation and fusion applications. |
| title | A Lipschitz spaces view of infinitely wide shallow neural networks |
| topic | Functional Analysis Machine Learning 68T07, 46E27, 46B20 |
| url | https://arxiv.org/abs/2410.14591 |