Koopman-based generalization bound: New aspect for full-rank weights
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929278404263936 |
|---|---|
| author | Hashimoto, Yuka Sonoda, Sho Ishikawa, Isao Nitanda, Atsushi Suzuki, Taiji |
| author_facet | Hashimoto, Yuka Sonoda, Sho Ishikawa, Isao Nitanda, Atsushi Suzuki, Taiji |
| contents | We propose a new bound for generalization of neural networks using Koopman operators. Whereas most of existing works focus on low-rank weight matrices, we focus on full-rank weight matrices. Our bound is tighter than existing norm-based bounds when the condition numbers of weight matrices are small. Especially, it is completely independent of the width of the network if the weight matrices are orthogonal. Our bound does not contradict to the existing bounds but is a complement to the existing bounds. As supported by several existing empirical results, low-rankness is not the only reason for generalization. Furthermore, our bound can be combined with the existing bounds to obtain a tighter bound. Our result sheds new light on understanding generalization of neural networks with full-rank weight matrices, and it provides a connection between operator-theoretic analysis and generalization of neural networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2302_05825 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Koopman-based generalization bound: New aspect for full-rank weights Hashimoto, Yuka Sonoda, Sho Ishikawa, Isao Nitanda, Atsushi Suzuki, Taiji Machine Learning Functional Analysis We propose a new bound for generalization of neural networks using Koopman operators. Whereas most of existing works focus on low-rank weight matrices, we focus on full-rank weight matrices. Our bound is tighter than existing norm-based bounds when the condition numbers of weight matrices are small. Especially, it is completely independent of the width of the network if the weight matrices are orthogonal. Our bound does not contradict to the existing bounds but is a complement to the existing bounds. As supported by several existing empirical results, low-rankness is not the only reason for generalization. Furthermore, our bound can be combined with the existing bounds to obtain a tighter bound. Our result sheds new light on understanding generalization of neural networks with full-rank weight matrices, and it provides a connection between operator-theoretic analysis and generalization of neural networks. |
| title | Koopman-based generalization bound: New aspect for full-rank weights |
| topic | Machine Learning Functional Analysis |
| url | https://arxiv.org/abs/2302.05825 |