From Linearity to Non-Linearity: How Masked Autoencoders Capture Spatial Correlations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914000302768128 |
|---|---|
| author | Bisulco, Anthony Ramesh, Rahul Balestriero, Randall Chaudhari, Pratik |
| author_facet | Bisulco, Anthony Ramesh, Rahul Balestriero, Randall Chaudhari, Pratik |
| contents | Masked Autoencoders (MAEs) have emerged as a powerful pretraining technique for vision foundation models. Despite their effectiveness, they require extensive hyperparameter tuning (masking ratio, patch size, encoder/decoder layers) when applied to novel datasets. While prior theoretical works have analyzed MAEs in terms of their attention patterns and hierarchical latent variable models, the connection between MAE hyperparameters and performance on downstream tasks is relatively unexplored. This work investigates how MAEs learn spatial correlations in the input image. We analytically derive the features learned by a linear MAE and show that masking ratio and patch size can be used to select for features that capture short- and long-range spatial correlations. We extend this analysis to non-linear MAEs to show that MAE representations adapt to spatial correlations in the dataset, beyond second-order statistics. Finally, we discuss some insights on how to select MAE hyper-parameters in practice. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_15404 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Linearity to Non-Linearity: How Masked Autoencoders Capture Spatial Correlations Bisulco, Anthony Ramesh, Rahul Balestriero, Randall Chaudhari, Pratik Computer Vision and Pattern Recognition Masked Autoencoders (MAEs) have emerged as a powerful pretraining technique for vision foundation models. Despite their effectiveness, they require extensive hyperparameter tuning (masking ratio, patch size, encoder/decoder layers) when applied to novel datasets. While prior theoretical works have analyzed MAEs in terms of their attention patterns and hierarchical latent variable models, the connection between MAE hyperparameters and performance on downstream tasks is relatively unexplored. This work investigates how MAEs learn spatial correlations in the input image. We analytically derive the features learned by a linear MAE and show that masking ratio and patch size can be used to select for features that capture short- and long-range spatial correlations. We extend this analysis to non-linear MAEs to show that MAE representations adapt to spatial correlations in the dataset, beyond second-order statistics. Finally, we discuss some insights on how to select MAE hyper-parameters in practice. |
| title | From Linearity to Non-Linearity: How Masked Autoencoders Capture Spatial Correlations |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2508.15404 |