Rolling the dice for better deep learning performance: A study of randomness techniques in deep neural networks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917630780112896 |
|---|---|
| author | Altarabichi, Mohammed Ghaith Nowaczyk, Sławomir Pashami, Sepideh Mashhadi, Peyman Sheikholharam Handl, Julia |
| author_facet | Altarabichi, Mohammed Ghaith Nowaczyk, Sławomir Pashami, Sepideh Mashhadi, Peyman Sheikholharam Handl, Julia |
| contents | This paper investigates how various randomization techniques impact Deep Neural Networks (DNNs). Randomization, like weight noise and dropout, aids in reducing overfitting and enhancing generalization, but their interactions are poorly understood. The study categorizes randomness techniques into four types and proposes new methods: adding noise to the loss function and random masking of gradient updates. Using Particle Swarm Optimizer (PSO) for hyperparameter optimization, it explores optimal configurations across MNIST, FASHION-MNIST, CIFAR10, and CIFAR100 datasets. Over 30,000 configurations are evaluated, revealing data augmentation and weight initialization randomness as main performance contributors. Correlation analysis shows different optimizers prefer distinct randomization types. The complete implementation and dataset are available on GitHub. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_03992 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Rolling the dice for better deep learning performance: A study of randomness techniques in deep neural networks Altarabichi, Mohammed Ghaith Nowaczyk, Sławomir Pashami, Sepideh Mashhadi, Peyman Sheikholharam Handl, Julia Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Neural and Evolutionary Computing This paper investigates how various randomization techniques impact Deep Neural Networks (DNNs). Randomization, like weight noise and dropout, aids in reducing overfitting and enhancing generalization, but their interactions are poorly understood. The study categorizes randomness techniques into four types and proposes new methods: adding noise to the loss function and random masking of gradient updates. Using Particle Swarm Optimizer (PSO) for hyperparameter optimization, it explores optimal configurations across MNIST, FASHION-MNIST, CIFAR10, and CIFAR100 datasets. Over 30,000 configurations are evaluated, revealing data augmentation and weight initialization randomness as main performance contributors. Correlation analysis shows different optimizers prefer distinct randomization types. The complete implementation and dataset are available on GitHub. |
| title | Rolling the dice for better deep learning performance: A study of randomness techniques in deep neural networks |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2404.03992 |