Optimizers Performance is Task-Dependent: An Empirical Study of Learning Rate Sensitivity in Classification and Regression Tasks
Fuente:
Zenodo
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Recurso digital |
| Sprache: | Altenglisch |
| Veröffentlicht: |
Zenodo
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866902213953060864 |
|---|---|
| author | Chisom ruth chibuike Kelechi Edwin Onuigbo Chika, Charles Ekene |
| author_facet | Chisom ruth chibuike Kelechi Edwin Onuigbo Chika, Charles Ekene |
| contents | <p>The performance of deep neural networks is critically influenced by the choice of optimization algorithm and its hyperparameters, particularly the learning rate (η). However, the interplay between optimizers and learning rates across different task paradigms, such as classification and regression, remains under-explored. This paper addresses this gap by presenting a systematic empirical study of four common Optimizers (Adaptive Moment Estimation (Adam), Adaptive Gradient Algorithm (Adagrad), Root Mean Square Propagation(RMSProp), and Stochastic Gradient Descent(SGD) with Nesterov Accelerated Gradient momentum) across a spectrum of learning rates . We evaluate 32 configurations on two distinct tasks: a Convolutional Neural Network (CNN) for image classification and a Multilayer Perceptron (MLP) for tabular regression. Our findings demonstrate that optimizer performance, stability, and learning rate sensitivity are highly task-dependent. We show that the optimal configuration diverges significantly between tasks: the CNN classification task achieved peak accuracy with RMSProp at a low η =10−4, whereas the MLP regression task achieved the highest R2 score (0.9102) with the non-adaptive SGD at a moderate η =10−2. These results provide strong empirical evidence that optimization strategies must be carefully tailored to the specific task and architecture.</p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_20392477 |
| institution | Zenodo |
| language | ang |
| publishDate | 2026 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Optimizers Performance is Task-Dependent: An Empirical Study of Learning Rate Sensitivity in Classification and Regression Tasks Chisom ruth chibuike Kelechi Edwin Onuigbo Chika, Charles Ekene Machine Learning, Deep Learning, Optimization, Learning Rates, Empirical Study, Convolutional Neural Networks, Multilayer Perceptron. <p>The performance of deep neural networks is critically influenced by the choice of optimization algorithm and its hyperparameters, particularly the learning rate (η). However, the interplay between optimizers and learning rates across different task paradigms, such as classification and regression, remains under-explored. This paper addresses this gap by presenting a systematic empirical study of four common Optimizers (Adaptive Moment Estimation (Adam), Adaptive Gradient Algorithm (Adagrad), Root Mean Square Propagation(RMSProp), and Stochastic Gradient Descent(SGD) with Nesterov Accelerated Gradient momentum) across a spectrum of learning rates . We evaluate 32 configurations on two distinct tasks: a Convolutional Neural Network (CNN) for image classification and a Multilayer Perceptron (MLP) for tabular regression. Our findings demonstrate that optimizer performance, stability, and learning rate sensitivity are highly task-dependent. We show that the optimal configuration diverges significantly between tasks: the CNN classification task achieved peak accuracy with RMSProp at a low η =10−4, whereas the MLP regression task achieved the highest R2 score (0.9102) with the non-adaptive SGD at a moderate η =10−2. These results provide strong empirical evidence that optimization strategies must be carefully tailored to the specific task and architecture.</p> |
| title | Optimizers Performance is Task-Dependent: An Empirical Study of Learning Rate Sensitivity in Classification and Regression Tasks |
| topic | Machine Learning, Deep Learning, Optimization, Learning Rates, Empirical Study, Convolutional Neural Networks, Multilayer Perceptron. |
| url | https://doi.org/10.5281/zenodo.20392477 |