A Globally Convergent Algorithm for Neural Network Parameter Optimization Based on Difference-of-Convex Functions
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916092317794304 |
|---|---|
| author | Tschernutter, Daniel Kraus, Mathias Feuerriegel, Stefan |
| author_facet | Tschernutter, Daniel Kraus, Mathias Feuerriegel, Stefan |
| contents | We propose an algorithm for optimizing the parameters of single hidden layer neural networks. Specifically, we derive a blockwise difference-of-convex (DC) functions representation of the objective function. Based on the latter, we propose a block coordinate descent (BCD) approach that we combine with a tailored difference-of-convex functions algorithm (DCA). We prove global convergence of the proposed algorithm. Furthermore, we mathematically analyze the convergence rate of parameters and the convergence rate in value (i.e., the training loss). We give conditions under which our algorithm converges linearly or even faster depending on the local shape of the loss function. We confirm our theoretical derivations numerically and compare our algorithm against state-of-the-art gradient-based solvers in terms of both training loss and test loss. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_07936 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Globally Convergent Algorithm for Neural Network Parameter Optimization Based on Difference-of-Convex Functions Tschernutter, Daniel Kraus, Mathias Feuerriegel, Stefan Machine Learning Neural and Evolutionary Computing Optimization and Control We propose an algorithm for optimizing the parameters of single hidden layer neural networks. Specifically, we derive a blockwise difference-of-convex (DC) functions representation of the objective function. Based on the latter, we propose a block coordinate descent (BCD) approach that we combine with a tailored difference-of-convex functions algorithm (DCA). We prove global convergence of the proposed algorithm. Furthermore, we mathematically analyze the convergence rate of parameters and the convergence rate in value (i.e., the training loss). We give conditions under which our algorithm converges linearly or even faster depending on the local shape of the loss function. We confirm our theoretical derivations numerically and compare our algorithm against state-of-the-art gradient-based solvers in terms of both training loss and test loss. |
| title | A Globally Convergent Algorithm for Neural Network Parameter Optimization Based on Difference-of-Convex Functions |
| topic | Machine Learning Neural and Evolutionary Computing Optimization and Control |
| url | https://arxiv.org/abs/2401.07936 |