A Survey of the Self Supervised Learning Mechanisms for Vision Transformers
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912552312635392 |
|---|---|
| author | Khan, Asifullah Sohail, Anabia Fiaz, Mustansar Hassan, Mehdi Afridi, Tariq Habib Marwat, Sibghat Ullah Munir, Farzeen Ali, Safdar Naseem, Hannan Zaheer, Muhammad Zaigham Ali, Kamran Sultana, Tangina Tanoli, Ziaurrehman Akhter, Naeem |
| author_facet | Khan, Asifullah Sohail, Anabia Fiaz, Mustansar Hassan, Mehdi Afridi, Tariq Habib Marwat, Sibghat Ullah Munir, Farzeen Ali, Safdar Naseem, Hannan Zaheer, Muhammad Zaigham Ali, Kamran Sultana, Tangina Tanoli, Ziaurrehman Akhter, Naeem |
| contents | Advances in deep learning are re-defining how visual data is processed and understand by the machines. Vision Transformers (ViTs) have recently demonstrated prominent performance in computer vision related tasks. However, their performance improves with increasing numbers of labeled data, indicating reliance on labeled data. Humanly annotated data are difficult to acquire and thus shifted the focus from traditional annotations to unsupervised learning strategies that learn structures inside the data. In response to this challenge, self-supervised learning (SSL) has emerged as a promising technique. SSL utilize inherent relationships within the data as a form of supervision. This technique can reduce the dependence on manual annotations and offers a more scalable and resource-effective approach to training models. Taking these strengths into account, it is necessary to assess the combination of SSL methods with ViTs, especially in the cases of limited labeled data. Inspired by this evolving trend, this survey aims to systematically review SSL mechanisms tailored for ViTs. We propose a comprehensive taxonomy to classify SSL techniques based on their representations and pre-training tasks. Furthermore, we highlighted the motivations behind the study of SSL, reviewed prominent pre-training tasks, and highlight advancements and challenges in this field. Furthermore, we conduct a comparative analysis of various SSL methods designed for ViTs, evaluating their strengths, limitations, and applicability to different scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_17059 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Survey of the Self Supervised Learning Mechanisms for Vision Transformers Khan, Asifullah Sohail, Anabia Fiaz, Mustansar Hassan, Mehdi Afridi, Tariq Habib Marwat, Sibghat Ullah Munir, Farzeen Ali, Safdar Naseem, Hannan Zaheer, Muhammad Zaigham Ali, Kamran Sultana, Tangina Tanoli, Ziaurrehman Akhter, Naeem Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Advances in deep learning are re-defining how visual data is processed and understand by the machines. Vision Transformers (ViTs) have recently demonstrated prominent performance in computer vision related tasks. However, their performance improves with increasing numbers of labeled data, indicating reliance on labeled data. Humanly annotated data are difficult to acquire and thus shifted the focus from traditional annotations to unsupervised learning strategies that learn structures inside the data. In response to this challenge, self-supervised learning (SSL) has emerged as a promising technique. SSL utilize inherent relationships within the data as a form of supervision. This technique can reduce the dependence on manual annotations and offers a more scalable and resource-effective approach to training models. Taking these strengths into account, it is necessary to assess the combination of SSL methods with ViTs, especially in the cases of limited labeled data. Inspired by this evolving trend, this survey aims to systematically review SSL mechanisms tailored for ViTs. We propose a comprehensive taxonomy to classify SSL techniques based on their representations and pre-training tasks. Furthermore, we highlighted the motivations behind the study of SSL, reviewed prominent pre-training tasks, and highlight advancements and challenges in this field. Furthermore, we conduct a comparative analysis of various SSL methods designed for ViTs, evaluating their strengths, limitations, and applicability to different scenarios. |
| title | A Survey of the Self Supervised Learning Mechanisms for Vision Transformers |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2408.17059 |