Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929528579817472 |
|---|---|
| author | Yang, Yahong Chen, Qipin Hao, Wenrui |
| author_facet | Yang, Yahong Chen, Qipin Hao, Wenrui |
| contents | In this paper, we present a novel training approach called the Homotopy Relaxation Training Algorithm (HRTA), aimed at accelerating the training process in contrast to traditional methods. Our algorithm incorporates two key mechanisms: one involves building a homotopy activation function that seamlessly connects the linear activation function with the ReLU activation function; the other technique entails relaxing the homotopy parameter to enhance the training refinement process. We have conducted an in-depth analysis of this novel method within the context of the neural tangent kernel (NTK), revealing significantly improved convergence rates. Our experimental results, especially when considering networks with larger widths, validate the theoretical conclusions. This proposed HRTA exhibits the potential for other activation functions and deep neural networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_15244 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks Yang, Yahong Chen, Qipin Hao, Wenrui Machine Learning Optimization and Control In this paper, we present a novel training approach called the Homotopy Relaxation Training Algorithm (HRTA), aimed at accelerating the training process in contrast to traditional methods. Our algorithm incorporates two key mechanisms: one involves building a homotopy activation function that seamlessly connects the linear activation function with the ReLU activation function; the other technique entails relaxing the homotopy parameter to enhance the training refinement process. We have conducted an in-depth analysis of this novel method within the context of the neural tangent kernel (NTK), revealing significantly improved convergence rates. Our experimental results, especially when considering networks with larger widths, validate the theoretical conclusions. This proposed HRTA exhibits the potential for other activation functions and deep neural networks. |
| title | Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2309.15244 |