Suppressing Non-Semantic Noise in Masked Image Modeling Representations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912993499938816 |
|---|---|
| author | Hjelkrem-Tan, Martine Aasan, Marius Chakraborty, Rwiddhi Arteaga, Gabriel Y. Choi, Changkyu Rivera, Adín Ramírez |
| author_facet | Hjelkrem-Tan, Martine Aasan, Marius Chakraborty, Rwiddhi Arteaga, Gabriel Y. Choi, Changkyu Rivera, Adín Ramírez |
| contents | Masked Image Modeling (MIM) has become a ubiquitous self-supervised vision paradigm. In this work, we show that MIM objectives cause the learned representations to retain non-semantic information, which ultimately hurts performance during inference. We introduce a model-agnostic score for semantic invariance using Principal Component Analysis (PCA) on real and synthetic non-semantic images. Based on this score, we propose a simple method, Semantically Orthogonal Artifact Projection (SOAP), to directly suppress non-semantic information in patch representations, leading to consistent improvements in zero-shot performance across various MIM-based models. SOAP is a post-hoc suppression method, requires zero training, and can be attached to any model as a single linear head. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_00172 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Suppressing Non-Semantic Noise in Masked Image Modeling Representations Hjelkrem-Tan, Martine Aasan, Marius Chakraborty, Rwiddhi Arteaga, Gabriel Y. Choi, Changkyu Rivera, Adín Ramírez Computer Vision and Pattern Recognition Masked Image Modeling (MIM) has become a ubiquitous self-supervised vision paradigm. In this work, we show that MIM objectives cause the learned representations to retain non-semantic information, which ultimately hurts performance during inference. We introduce a model-agnostic score for semantic invariance using Principal Component Analysis (PCA) on real and synthetic non-semantic images. Based on this score, we propose a simple method, Semantically Orthogonal Artifact Projection (SOAP), to directly suppress non-semantic information in patch representations, leading to consistent improvements in zero-shot performance across various MIM-based models. SOAP is a post-hoc suppression method, requires zero training, and can be attached to any model as a single linear head. |
| title | Suppressing Non-Semantic Noise in Masked Image Modeling Representations |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2604.00172 |