An Augmented Backward-Corrected Projector Splitting Integrator for Dynamical Low-Rank Training
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909477660262400 |
|---|---|
| author | Kusch, Jonas Schotthöfer, Steffen Walter, Alexandra |
| author_facet | Kusch, Jonas Schotthöfer, Steffen Walter, Alexandra |
| contents | Layer factorization has emerged as a widely used technique for training memory-efficient neural networks. However, layer factorization methods face several challenges, particularly a lack of robustness during the training process. To overcome this limitation, dynamical low-rank training methods have been developed, utilizing robust time integration techniques for low-rank matrix differential equations. Although these approaches facilitate efficient training, they still depend on computationally intensive QR and singular value decompositions of matrices with small rank. In this work, we introduce a novel low-rank training method that reduces the number of required QR decompositions. Our approach integrates an augmentation step into a projector-splitting scheme, ensuring convergence to a locally optimal solution. We provide a rigorous theoretical analysis of the proposed method and demonstrate its effectiveness across multiple benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_03006 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | An Augmented Backward-Corrected Projector Splitting Integrator for Dynamical Low-Rank Training Kusch, Jonas Schotthöfer, Steffen Walter, Alexandra Numerical Analysis Machine Learning Layer factorization has emerged as a widely used technique for training memory-efficient neural networks. However, layer factorization methods face several challenges, particularly a lack of robustness during the training process. To overcome this limitation, dynamical low-rank training methods have been developed, utilizing robust time integration techniques for low-rank matrix differential equations. Although these approaches facilitate efficient training, they still depend on computationally intensive QR and singular value decompositions of matrices with small rank. In this work, we introduce a novel low-rank training method that reduces the number of required QR decompositions. Our approach integrates an augmentation step into a projector-splitting scheme, ensuring convergence to a locally optimal solution. We provide a rigorous theoretical analysis of the proposed method and demonstrate its effectiveness across multiple benchmarks. |
| title | An Augmented Backward-Corrected Projector Splitting Integrator for Dynamical Low-Rank Training |
| topic | Numerical Analysis Machine Learning |
| url | https://arxiv.org/abs/2502.03006 |