Spectral State Space Model for Rotation-Invariant Visual Representation Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912285340991488 |
|---|---|
| author | Dastani, Sahar Bahri, Ali Yazdanpanah, Moslem Noori, Mehrdad Osowiechi, David Hakim, Gustavo Adolfo Vargas Beizaee, Farzad Cheraghalikhani, Milad Mondal, Arnab Kumar Lombaert, Herve Desrosiers, Christian |
| author_facet | Dastani, Sahar Bahri, Ali Yazdanpanah, Moslem Noori, Mehrdad Osowiechi, David Hakim, Gustavo Adolfo Vargas Beizaee, Farzad Cheraghalikhani, Milad Mondal, Arnab Kumar Lombaert, Herve Desrosiers, Christian |
| contents | State Space Models (SSMs) have recently emerged as an alternative to Vision Transformers (ViTs) due to their unique ability of modeling global relationships with linear complexity. SSMs are specifically designed to capture spatially proximate relationships of image patches. However, they fail to identify relationships between conceptually related yet not adjacent patches. This limitation arises from the non-causal nature of image data, which lacks inherent directional relationships. Additionally, current vision-based SSMs are highly sensitive to transformations such as rotation. Their predefined scanning directions depend on the original image orientation, which can cause the model to produce inconsistent patch-processing sequences after rotation. To address these limitations, we introduce Spectral VMamba, a novel approach that effectively captures the global structure within an image by leveraging spectral information derived from the graph Laplacian of image patches. Through spectral decomposition, our approach encodes patch relationships independently of image orientation, achieving rotation invariance with the aid of our Rotational Feature Normalizer (RFN) module. Our experiments on classification tasks show that Spectral VMamba outperforms the leading SSM models in vision, such as VMamba, while maintaining invariance to rotations and a providing a similar runtime efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_06369 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Spectral State Space Model for Rotation-Invariant Visual Representation Learning Dastani, Sahar Bahri, Ali Yazdanpanah, Moslem Noori, Mehrdad Osowiechi, David Hakim, Gustavo Adolfo Vargas Beizaee, Farzad Cheraghalikhani, Milad Mondal, Arnab Kumar Lombaert, Herve Desrosiers, Christian Computer Vision and Pattern Recognition State Space Models (SSMs) have recently emerged as an alternative to Vision Transformers (ViTs) due to their unique ability of modeling global relationships with linear complexity. SSMs are specifically designed to capture spatially proximate relationships of image patches. However, they fail to identify relationships between conceptually related yet not adjacent patches. This limitation arises from the non-causal nature of image data, which lacks inherent directional relationships. Additionally, current vision-based SSMs are highly sensitive to transformations such as rotation. Their predefined scanning directions depend on the original image orientation, which can cause the model to produce inconsistent patch-processing sequences after rotation. To address these limitations, we introduce Spectral VMamba, a novel approach that effectively captures the global structure within an image by leveraging spectral information derived from the graph Laplacian of image patches. Through spectral decomposition, our approach encodes patch relationships independently of image orientation, achieving rotation invariance with the aid of our Rotational Feature Normalizer (RFN) module. Our experiments on classification tasks show that Spectral VMamba outperforms the leading SSM models in vision, such as VMamba, while maintaining invariance to rotations and a providing a similar runtime efficiency. |
| title | Spectral State Space Model for Rotation-Invariant Visual Representation Learning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.06369 |