Are Sparse Neural Networks Better Hard Sample Learners?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915080036155392 |
|---|---|
| author | Xiao, Qiao Wu, Boqian Yin, Lu Gadzinski, Christopher Neil Huang, Tianjin Pechenizkiy, Mykola Mocanu, Decebal Constantin |
| author_facet | Xiao, Qiao Wu, Boqian Yin, Lu Gadzinski, Christopher Neil Huang, Tianjin Pechenizkiy, Mykola Mocanu, Decebal Constantin |
| contents | While deep learning has demonstrated impressive progress, it remains a daunting challenge to learn from hard samples as these samples are usually noisy and intricate. These hard samples play a crucial role in the optimal performance of deep neural networks. Most research on Sparse Neural Networks (SNNs) has focused on standard training data, leaving gaps in understanding their effectiveness on complex and challenging data. This paper's extensive investigation across scenarios reveals that most SNNs trained on challenging samples can often match or surpass dense models in accuracy at certain sparsity levels, especially with limited data. We observe that layer-wise density ratios tend to play an important role in SNN performance, particularly for methods that train from scratch without pre-trained initialization. These insights enhance our understanding of SNNs' behavior and potential for efficient learning approaches in data-centric AI. Our code is publicly available at: \url{https://github.com/QiaoXiao7282/hard_sample_learners}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_09196 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Are Sparse Neural Networks Better Hard Sample Learners? Xiao, Qiao Wu, Boqian Yin, Lu Gadzinski, Christopher Neil Huang, Tianjin Pechenizkiy, Mykola Mocanu, Decebal Constantin Computer Vision and Pattern Recognition Machine Learning While deep learning has demonstrated impressive progress, it remains a daunting challenge to learn from hard samples as these samples are usually noisy and intricate. These hard samples play a crucial role in the optimal performance of deep neural networks. Most research on Sparse Neural Networks (SNNs) has focused on standard training data, leaving gaps in understanding their effectiveness on complex and challenging data. This paper's extensive investigation across scenarios reveals that most SNNs trained on challenging samples can often match or surpass dense models in accuracy at certain sparsity levels, especially with limited data. We observe that layer-wise density ratios tend to play an important role in SNN performance, particularly for methods that train from scratch without pre-trained initialization. These insights enhance our understanding of SNNs' behavior and potential for efficient learning approaches in data-centric AI. Our code is publicly available at: \url{https://github.com/QiaoXiao7282/hard_sample_learners}. |
| title | Are Sparse Neural Networks Better Hard Sample Learners? |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2409.09196 |