Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.00538 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909446182010880 |
|---|---|
| author | Hasan, Md. Tarek Akter, Arifa Shamael, Mohammad Nazmush Hossain, Md Al Emran Billah, H. M. Mutasim Islam, Sumayra Shatabda, Swakkhar |
| author_facet | Hasan, Md. Tarek Akter, Arifa Shamael, Mohammad Nazmush Hossain, Md Al Emran Billah, H. M. Mutasim Islam, Sumayra Shatabda, Swakkhar |
| contents | Dropout is an effective strategy for the regularization of deep neural networks. Applying tabu to the units that have been dropped in the recent epoch and retaining them for training ensures diversification in dropout. In this paper, we improve the Tabu Dropout mechanism for training deep neural networks in two ways. Firstly, we propose to use tabu tenure, or the number of epochs a particular unit will not be dropped. Different tabu tenures provide diversification to boost the training of deep neural networks based on the search landscape. Secondly, we propose an adaptive tabu algorithm that automatically selects the tabu tenure based on the training performances through epochs. On several standard benchmark datasets, the experimental results show that the adaptive tabu dropout and tabu tenure dropout diversify and perform significantly better compared to the standard dropout and basic tabu dropout mechanisms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_00538 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Adaptive Tabu Dropout for Regularization of Deep Neural Network Hasan, Md. Tarek Akter, Arifa Shamael, Mohammad Nazmush Hossain, Md Al Emran Billah, H. M. Mutasim Islam, Sumayra Shatabda, Swakkhar Machine Learning Dropout is an effective strategy for the regularization of deep neural networks. Applying tabu to the units that have been dropped in the recent epoch and retaining them for training ensures diversification in dropout. In this paper, we improve the Tabu Dropout mechanism for training deep neural networks in two ways. Firstly, we propose to use tabu tenure, or the number of epochs a particular unit will not be dropped. Different tabu tenures provide diversification to boost the training of deep neural networks based on the search landscape. Secondly, we propose an adaptive tabu algorithm that automatically selects the tabu tenure based on the training performances through epochs. On several standard benchmark datasets, the experimental results show that the adaptive tabu dropout and tabu tenure dropout diversify and perform significantly better compared to the standard dropout and basic tabu dropout mechanisms. |
| title | Adaptive Tabu Dropout for Regularization of Deep Neural Network |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2501.00538 |