SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connection
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913212763471872 |
|---|---|
| author | Ataiefard, Foozhan Ahmed, Walid Hajimolahoseini, Habib Asani, Saina Javadi, Farnoosh Hassanpour, Mohammad Awad, Omar Mohamed Wen, Austin Liu, Kangling Liu, Yang |
| author_facet | Ataiefard, Foozhan Ahmed, Walid Hajimolahoseini, Habib Asani, Saina Javadi, Farnoosh Hassanpour, Mohammad Awad, Omar Mohamed Wen, Austin Liu, Kangling Liu, Yang |
| contents | Vision transformers are known to be more computationally and data-intensive than CNN models. These transformer models such as ViT, require all the input image tokens to learn the relationship among them. However, many of these tokens are not informative and may contain irrelevant information such as unrelated background or unimportant scenery. These tokens are overlooked by the multi-head self-attention (MHSA), resulting in many redundant and unnecessary computations in MHSA and the feed-forward network (FFN). In this work, we propose a method to optimize the amount of unnecessary interactions between unimportant tokens by separating and sending them through a different low-cost computational path. Our method does not add any parameters to the ViT model and aims to find the best trade-off between training throughput and achieving a 0% loss in the Top-1 accuracy of the final model. Our experimental results on training ViT-small from scratch show that SkipViT is capable of effectively dropping 55% of the tokens while gaining more than 13% training throughput and maintaining classification accuracy at the level of the baseline model on Huawei Ascend910A. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_15293 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connection Ataiefard, Foozhan Ahmed, Walid Hajimolahoseini, Habib Asani, Saina Javadi, Farnoosh Hassanpour, Mohammad Awad, Omar Mohamed Wen, Austin Liu, Kangling Liu, Yang Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Vision transformers are known to be more computationally and data-intensive than CNN models. These transformer models such as ViT, require all the input image tokens to learn the relationship among them. However, many of these tokens are not informative and may contain irrelevant information such as unrelated background or unimportant scenery. These tokens are overlooked by the multi-head self-attention (MHSA), resulting in many redundant and unnecessary computations in MHSA and the feed-forward network (FFN). In this work, we propose a method to optimize the amount of unnecessary interactions between unimportant tokens by separating and sending them through a different low-cost computational path. Our method does not add any parameters to the ViT model and aims to find the best trade-off between training throughput and achieving a 0% loss in the Top-1 accuracy of the final model. Our experimental results on training ViT-small from scratch show that SkipViT is capable of effectively dropping 55% of the tokens while gaining more than 13% training throughput and maintaining classification accuracy at the level of the baseline model on Huawei Ascend910A. |
| title | SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connection |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2401.15293 |