AdamZ: An Enhanced Optimisation Method for Neural Network Training
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917847387602944 |
|---|---|
| author | Zaznov, Ilia Badii, Atta Dufour, Alfonso Kunkel, Julian |
| author_facet | Zaznov, Ilia Badii, Atta Dufour, Alfonso Kunkel, Julian |
| contents | AdamZ is an advanced variant of the Adam optimiser, developed to enhance convergence efficiency in neural network training. This optimiser dynamically adjusts the learning rate by incorporating mechanisms to address overshooting and stagnation, that are common challenges in optimisation. Specifically, AdamZ reduces the learning rate when overshooting is detected and increases it during periods of stagnation, utilising hyperparameters such as overshoot and stagnation factors, thresholds, and patience levels to guide these adjustments. While AdamZ may lead to slightly longer training times compared to some other optimisers, it consistently excels in minimising the loss function, making it particularly advantageous for applications where precision is critical. Benchmarking results demonstrate the effectiveness of AdamZ in maintaining optimal learning rates, leading to improved model performance across diverse tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_15375 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AdamZ: An Enhanced Optimisation Method for Neural Network Training Zaznov, Ilia Badii, Atta Dufour, Alfonso Kunkel, Julian Machine Learning Artificial Intelligence Neural and Evolutionary Computing Optimization and Control AdamZ is an advanced variant of the Adam optimiser, developed to enhance convergence efficiency in neural network training. This optimiser dynamically adjusts the learning rate by incorporating mechanisms to address overshooting and stagnation, that are common challenges in optimisation. Specifically, AdamZ reduces the learning rate when overshooting is detected and increases it during periods of stagnation, utilising hyperparameters such as overshoot and stagnation factors, thresholds, and patience levels to guide these adjustments. While AdamZ may lead to slightly longer training times compared to some other optimisers, it consistently excels in minimising the loss function, making it particularly advantageous for applications where precision is critical. Benchmarking results demonstrate the effectiveness of AdamZ in maintaining optimal learning rates, leading to improved model performance across diverse tasks. |
| title | AdamZ: An Enhanced Optimisation Method for Neural Network Training |
| topic | Machine Learning Artificial Intelligence Neural and Evolutionary Computing Optimization and Control |
| url | https://arxiv.org/abs/2411.15375 |