Autoregressive Pretraining with Mamba in Vision
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910482352308224 |
|---|---|
| author | Ren, Sucheng Li, Xianhang Tu, Haoqin Wang, Feng Shu, Fangxun Zhang, Lei Mei, Jieru Yang, Linjie Wang, Peng Wang, Heng Yuille, Alan Xie, Cihang |
| author_facet | Ren, Sucheng Li, Xianhang Tu, Haoqin Wang, Feng Shu, Fangxun Zhang, Lei Mei, Jieru Yang, Linjie Wang, Peng Wang, Heng Yuille, Alan Xie, Cihang |
| contents | The vision community has started to build with the recently developed state space model, Mamba, as the new backbone for a range of tasks. This paper shows that Mamba's visual capability can be significantly enhanced through autoregressive pretraining, a direction not previously explored. Efficiency-wise, the autoregressive nature can well capitalize on the Mamba's unidirectional recurrent structure, enabling faster overall training speed compared to other training strategies like mask modeling. Performance-wise, autoregressive pretraining equips the Mamba architecture with markedly higher accuracy over its supervised-trained counterparts and, more importantly, successfully unlocks its scaling potential to large and even huge model sizes. For example, with autoregressive pretraining, a base-size Mamba attains 83.2\% ImageNet accuracy, outperforming its supervised counterpart by 2.0\%; our huge-size Mamba, the largest Vision Mamba to date, attains 85.0\% ImageNet accuracy (85.5\% when finetuned with $384\times384$ inputs), notably surpassing all other Mamba variants in vision. The code is available at \url{https://github.com/OliverRensu/ARM}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07537 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Autoregressive Pretraining with Mamba in Vision Ren, Sucheng Li, Xianhang Tu, Haoqin Wang, Feng Shu, Fangxun Zhang, Lei Mei, Jieru Yang, Linjie Wang, Peng Wang, Heng Yuille, Alan Xie, Cihang Computer Vision and Pattern Recognition The vision community has started to build with the recently developed state space model, Mamba, as the new backbone for a range of tasks. This paper shows that Mamba's visual capability can be significantly enhanced through autoregressive pretraining, a direction not previously explored. Efficiency-wise, the autoregressive nature can well capitalize on the Mamba's unidirectional recurrent structure, enabling faster overall training speed compared to other training strategies like mask modeling. Performance-wise, autoregressive pretraining equips the Mamba architecture with markedly higher accuracy over its supervised-trained counterparts and, more importantly, successfully unlocks its scaling potential to large and even huge model sizes. For example, with autoregressive pretraining, a base-size Mamba attains 83.2\% ImageNet accuracy, outperforming its supervised counterpart by 2.0\%; our huge-size Mamba, the largest Vision Mamba to date, attains 85.0\% ImageNet accuracy (85.5\% when finetuned with $384\times384$ inputs), notably surpassing all other Mamba variants in vision. The code is available at \url{https://github.com/OliverRensu/ARM}. |
| title | Autoregressive Pretraining with Mamba in Vision |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.07537 |