Deep Learning Optimization Using Self-Adaptive Weighted Auxiliary Variables
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912354752528384 |
|---|---|
| author | Liu, Yaru Gu, Yiqi Ng, Michael K. |
| author_facet | Liu, Yaru Gu, Yiqi Ng, Michael K. |
| contents | In this paper, we develop a new optimization framework for the least squares learning problem via fully connected neural networks or physics-informed neural networks. The gradient descent sometimes behaves inefficiently in deep learning because of the high non-convexity of loss functions and the vanishing gradient issue. Our idea is to introduce auxiliary variables to separate the layers of the deep neural networks and reformulate the loss functions for ease of optimization. We design the self-adaptive weights to preserve the consistency between the reformulated loss and the original mean squared loss, which guarantees that optimizing the new loss helps optimize the original problem. Numerical experiments are presented to verify the consistency and show the effectiveness and robustness of our models over gradient descent. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_21501 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Deep Learning Optimization Using Self-Adaptive Weighted Auxiliary Variables Liu, Yaru Gu, Yiqi Ng, Michael K. Machine Learning In this paper, we develop a new optimization framework for the least squares learning problem via fully connected neural networks or physics-informed neural networks. The gradient descent sometimes behaves inefficiently in deep learning because of the high non-convexity of loss functions and the vanishing gradient issue. Our idea is to introduce auxiliary variables to separate the layers of the deep neural networks and reformulate the loss functions for ease of optimization. We design the self-adaptive weights to preserve the consistency between the reformulated loss and the original mean squared loss, which guarantees that optimizing the new loss helps optimize the original problem. Numerical experiments are presented to verify the consistency and show the effectiveness and robustness of our models over gradient descent. |
| title | Deep Learning Optimization Using Self-Adaptive Weighted Auxiliary Variables |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2504.21501 |