Optimization Dynamics of Equivariant and Augmented Neural Networks
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914976949600256 |
|---|---|
| author | Nordenfors, Oskar Ohlsson, Fredrik Flinth, Axel |
| author_facet | Nordenfors, Oskar Ohlsson, Fredrik Flinth, Axel |
| contents | We investigate the optimization of neural networks on symmetric data, and compare the strategy of constraining the architecture to be equivariant to that of using data augmentation. Our analysis reveals that that the relative geometry of the admissible and the equivariant layers, respectively, plays a key role. Under natural assumptions on the data, network, loss, and group of symmetries, we show that compatibility of the spaces of admissible layers and equivariant layers, in the sense that the corresponding orthogonal projections commute, implies that the sets of equivariant stationary points are identical for the two strategies. If the linear layers of the network also are given a unitary parametrization, the set of equivariant layers is even invariant under the gradient flow for augmented models. Our analysis however also reveals that even in the latter situation, stationary points may be unstable for augmented training although they are stable for the manifestly equivariant models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2303_13458 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Optimization Dynamics of Equivariant and Augmented Neural Networks Nordenfors, Oskar Ohlsson, Fredrik Flinth, Axel Machine Learning Optimization and Control 68T07, 20C35, 37N40 We investigate the optimization of neural networks on symmetric data, and compare the strategy of constraining the architecture to be equivariant to that of using data augmentation. Our analysis reveals that that the relative geometry of the admissible and the equivariant layers, respectively, plays a key role. Under natural assumptions on the data, network, loss, and group of symmetries, we show that compatibility of the spaces of admissible layers and equivariant layers, in the sense that the corresponding orthogonal projections commute, implies that the sets of equivariant stationary points are identical for the two strategies. If the linear layers of the network also are given a unitary parametrization, the set of equivariant layers is even invariant under the gradient flow for augmented models. Our analysis however also reveals that even in the latter situation, stationary points may be unstable for augmented training although they are stable for the manifestly equivariant models. |
| title | Optimization Dynamics of Equivariant and Augmented Neural Networks |
| topic | Machine Learning Optimization and Control 68T07, 20C35, 37N40 |
| url | https://arxiv.org/abs/2303.13458 |