Efficient Finite Initialization with Partial Norms for Tensorized Neural Networks and Tensor Networks Algorithms
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914522012319744 |
|---|---|
| author | Ali, Alejandro Mata Delgado, Iñigo Perez Roura, Marina Ristol de Leceta, Aitor Moreno Fdez. |
| author_facet | Ali, Alejandro Mata Delgado, Iñigo Perez Roura, Marina Ristol de Leceta, Aitor Moreno Fdez. |
| contents | We present two algorithms to initialize layers of tensorized neural networks and general tensor network algorithms using partial computations of their Frobenius norms and positive lineal entrywise sums, depending on the type of tensor network involved. The core of this method is the use of the norm of subnetworks of the tensor network in an iterative way, so that we normalize by the finite values of the norms that led to the divergence or zero norm. In addition, the method benefits from the reuse of intermediate calculations. We have also applied it to the Matrix Product State/Tensor Train (MPS/TT) and Matrix Product Operator/Tensor Train Matrix (MPO/TT-M) layers and have seen its scaling versus the number of nodes, bond dimension, and physical dimension. All code is publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_06577 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Efficient Finite Initialization with Partial Norms for Tensorized Neural Networks and Tensor Networks Algorithms Ali, Alejandro Mata Delgado, Iñigo Perez Roura, Marina Ristol de Leceta, Aitor Moreno Fdez. Machine Learning Quantum Physics 68Q12, 15A69, 68T07 We present two algorithms to initialize layers of tensorized neural networks and general tensor network algorithms using partial computations of their Frobenius norms and positive lineal entrywise sums, depending on the type of tensor network involved. The core of this method is the use of the norm of subnetworks of the tensor network in an iterative way, so that we normalize by the finite values of the norms that led to the divergence or zero norm. In addition, the method benefits from the reuse of intermediate calculations. We have also applied it to the Matrix Product State/Tensor Train (MPS/TT) and Matrix Product Operator/Tensor Train Matrix (MPO/TT-M) layers and have seen its scaling versus the number of nodes, bond dimension, and physical dimension. All code is publicly available. |
| title | Efficient Finite Initialization with Partial Norms for Tensorized Neural Networks and Tensor Networks Algorithms |
| topic | Machine Learning Quantum Physics 68Q12, 15A69, 68T07 |
| url | https://arxiv.org/abs/2309.06577 |