Dynamic Vision Mamba
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916676195319808 |
|---|---|
| author | Wu, Mengxuan Li, Zekai Liang, Zhiyuan Li, Moyang Zhao, Xuanlei Khaki, Samir Zhu, Zheng Peng, Xiaojiang Plataniotis, Konstantinos N. Wang, Kai Zhao, Wangbo You, Yang |
| author_facet | Wu, Mengxuan Li, Zekai Liang, Zhiyuan Li, Moyang Zhao, Xuanlei Khaki, Samir Zhu, Zheng Peng, Xiaojiang Plataniotis, Konstantinos N. Wang, Kai Zhao, Wangbo You, Yang |
| contents | Mamba-based vision models have gained extensive attention as a result of being computationally more efficient than attention-based models. However, spatial redundancy still exists in these models, represented by token and block redundancy. For token redundancy, we analytically find that early token pruning methods will result in inconsistency between training and inference or introduce extra computation for inference. Therefore, we customize token pruning to fit the Mamba structure by rearranging the pruned sequence before feeding it into the next Mamba block. For block redundancy, we allow each image to select SSM blocks dynamically based on an empirical observation that the inference speed of Mamba-based vision models is largely affected by the number of SSM blocks. Our proposed method, Dynamic Vision Mamba (DyVM), effectively reduces FLOPs with minor performance drops. We achieve a reduction of 35.2\% FLOPs with only a loss of accuracy of 1.7\% on Vim-S. It also generalizes well across different Mamba vision model architectures and different vision tasks. Our code will be made public. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_04787 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dynamic Vision Mamba Wu, Mengxuan Li, Zekai Liang, Zhiyuan Li, Moyang Zhao, Xuanlei Khaki, Samir Zhu, Zheng Peng, Xiaojiang Plataniotis, Konstantinos N. Wang, Kai Zhao, Wangbo You, Yang Computer Vision and Pattern Recognition Artificial Intelligence Mamba-based vision models have gained extensive attention as a result of being computationally more efficient than attention-based models. However, spatial redundancy still exists in these models, represented by token and block redundancy. For token redundancy, we analytically find that early token pruning methods will result in inconsistency between training and inference or introduce extra computation for inference. Therefore, we customize token pruning to fit the Mamba structure by rearranging the pruned sequence before feeding it into the next Mamba block. For block redundancy, we allow each image to select SSM blocks dynamically based on an empirical observation that the inference speed of Mamba-based vision models is largely affected by the number of SSM blocks. Our proposed method, Dynamic Vision Mamba (DyVM), effectively reduces FLOPs with minor performance drops. We achieve a reduction of 35.2\% FLOPs with only a loss of accuracy of 1.7\% on Vim-S. It also generalizes well across different Mamba vision model architectures and different vision tasks. Our code will be made public. |
| title | Dynamic Vision Mamba |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2504.04787 |