Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911837443850240 |
|---|---|
| author | Choi, Hyesong Park, Hyejin Yi, Kwang Moo Cha, Sungmin Min, Dongbo |
| author_facet | Choi, Hyesong Park, Hyejin Yi, Kwang Moo Cha, Sungmin Min, Dongbo |
| contents | In this paper, we introduce Saliency-Based Adaptive Masking (SBAM), a novel and cost-effective approach that significantly enhances the pre-training performance of Masked Image Modeling (MIM) approaches by prioritizing token salience. Our method provides robustness against variations in masking ratios, effectively mitigating the performance instability issues common in existing methods. This relaxes the sensitivity of MIM-based pre-training to masking ratios, which in turn allows us to propose an adaptive strategy for `tailored' masking ratios for each data sample, which no existing method can provide. Toward this goal, we propose an Adaptive Masking Ratio (AMR) strategy that dynamically adjusts the proportion of masking for the unique content of each image based on token salience. We show that our method significantly improves over the state-of-the-art in mask-based pre-training on the ImageNet-1K dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_08327 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training Choi, Hyesong Park, Hyejin Yi, Kwang Moo Cha, Sungmin Min, Dongbo Computer Vision and Pattern Recognition In this paper, we introduce Saliency-Based Adaptive Masking (SBAM), a novel and cost-effective approach that significantly enhances the pre-training performance of Masked Image Modeling (MIM) approaches by prioritizing token salience. Our method provides robustness against variations in masking ratios, effectively mitigating the performance instability issues common in existing methods. This relaxes the sensitivity of MIM-based pre-training to masking ratios, which in turn allows us to propose an adaptive strategy for `tailored' masking ratios for each data sample, which no existing method can provide. Toward this goal, we propose an Adaptive Masking Ratio (AMR) strategy that dynamically adjusts the proportion of masking for the unique content of each image based on token salience. We show that our method significantly improves over the state-of-the-art in mask-based pre-training on the ImageNet-1K dataset. |
| title | Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.08327 |