FLAVARS: A Multimodal Foundational Language and Vision Alignment Model for Remote Sensing
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910785427472384 |
|---|---|
| author | Corley, Isaac Nsutezo, Simone Fobi Ortiz, Anthony Robinson, Caleb Dodhia, Rahul Ferres, Juan M. Lavista Najafirad, Peyman |
| author_facet | Corley, Isaac Nsutezo, Simone Fobi Ortiz, Anthony Robinson, Caleb Dodhia, Rahul Ferres, Juan M. Lavista Najafirad, Peyman |
| contents | Remote sensing imagery is dense with objects and contextual visual information. There is a recent trend to combine paired satellite images and text captions for pretraining performant encoders for downstream tasks. However, while contrastive image-text methods like CLIP enable vision-language alignment and zero-shot classification ability, vision-only downstream performance tends to degrade compared to image-only pretraining, such as MAE. In this paper, we propose FLAVARS, a pretraining method that combines the best of both contrastive learning and masked modeling, along with geospatial alignment via contrastive location encoding. We find that FLAVARS significantly outperforms a baseline of SkyCLIP for vision-only tasks such as KNN classification and semantic segmentation, +6\% mIOU on SpaceNet1, while retaining the ability to perform zero-shot classification, unlike MAE pretrained methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_08490 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FLAVARS: A Multimodal Foundational Language and Vision Alignment Model for Remote Sensing Corley, Isaac Nsutezo, Simone Fobi Ortiz, Anthony Robinson, Caleb Dodhia, Rahul Ferres, Juan M. Lavista Najafirad, Peyman Computer Vision and Pattern Recognition Machine Learning Remote sensing imagery is dense with objects and contextual visual information. There is a recent trend to combine paired satellite images and text captions for pretraining performant encoders for downstream tasks. However, while contrastive image-text methods like CLIP enable vision-language alignment and zero-shot classification ability, vision-only downstream performance tends to degrade compared to image-only pretraining, such as MAE. In this paper, we propose FLAVARS, a pretraining method that combines the best of both contrastive learning and masked modeling, along with geospatial alignment via contrastive location encoding. We find that FLAVARS significantly outperforms a baseline of SkyCLIP for vision-only tasks such as KNN classification and semantic segmentation, +6\% mIOU on SpaceNet1, while retaining the ability to perform zero-shot classification, unlike MAE pretrained methods. |
| title | FLAVARS: A Multimodal Foundational Language and Vision Alignment Model for Remote Sensing |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2501.08490 |