EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916675793715200 |
|---|---|
| author | Soni, Sagar Dudhane, Akshay Debary, Hiyam Fiaz, Mustansar Munir, Muhammad Akhtar Danish, Muhammad Sohail Fraccaro, Paolo Watson, Campbell D Klein, Levente J Khan, Fahad Shahbaz Khan, Salman |
| author_facet | Soni, Sagar Dudhane, Akshay Debary, Hiyam Fiaz, Mustansar Munir, Muhammad Akhtar Danish, Muhammad Sohail Fraccaro, Paolo Watson, Campbell D Klein, Levente J Khan, Fahad Shahbaz Khan, Salman |
| contents | Automated analysis of vast Earth observation data via interactive Vision-Language Models (VLMs) can unlock new opportunities for environmental monitoring, disaster response, and {resource management}. Existing generic VLMs do not perform well on Remote Sensing data, while the recent Geo-spatial VLMs remain restricted to a fixed resolution and few sensor modalities. In this paper, we introduce EarthDial, a conversational assistant specifically designed for Earth Observation (EO) data, transforming complex, multi-sensory Earth observations into interactive, natural language dialogues. EarthDial supports multi-spectral, multi-temporal, and multi-resolution imagery, enabling a wide range of remote sensing tasks, including classification, detection, captioning, question answering, visual reasoning, and visual grounding. To achieve this, we introduce an extensive instruction tuning dataset comprising over 11.11M instruction pairs covering RGB, Synthetic Aperture Radar (SAR), and multispectral modalities such as Near-Infrared (NIR) and infrared. Furthermore, EarthDial handles bi-temporal and multi-temporal sequence analysis for applications like change detection. Our extensive experimental results on 44 downstream datasets demonstrate that EarthDial outperforms existing generic and domain-specific models, achieving better generalization across various EO tasks. Our source codes and pre-trained models are at https://github.com/hiyamdebary/EarthDial. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_15190 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues Soni, Sagar Dudhane, Akshay Debary, Hiyam Fiaz, Mustansar Munir, Muhammad Akhtar Danish, Muhammad Sohail Fraccaro, Paolo Watson, Campbell D Klein, Levente J Khan, Fahad Shahbaz Khan, Salman Computer Vision and Pattern Recognition Automated analysis of vast Earth observation data via interactive Vision-Language Models (VLMs) can unlock new opportunities for environmental monitoring, disaster response, and {resource management}. Existing generic VLMs do not perform well on Remote Sensing data, while the recent Geo-spatial VLMs remain restricted to a fixed resolution and few sensor modalities. In this paper, we introduce EarthDial, a conversational assistant specifically designed for Earth Observation (EO) data, transforming complex, multi-sensory Earth observations into interactive, natural language dialogues. EarthDial supports multi-spectral, multi-temporal, and multi-resolution imagery, enabling a wide range of remote sensing tasks, including classification, detection, captioning, question answering, visual reasoning, and visual grounding. To achieve this, we introduce an extensive instruction tuning dataset comprising over 11.11M instruction pairs covering RGB, Synthetic Aperture Radar (SAR), and multispectral modalities such as Near-Infrared (NIR) and infrared. Furthermore, EarthDial handles bi-temporal and multi-temporal sequence analysis for applications like change detection. Our extensive experimental results on 44 downstream datasets demonstrate that EarthDial outperforms existing generic and domain-specific models, achieving better generalization across various EO tasks. Our source codes and pre-trained models are at https://github.com/hiyamdebary/EarthDial. |
| title | EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.15190 |