Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.06234 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913666278883328 |
|---|---|
| author | Irvin, Jeremy Andrew Liu, Emily Ruoyu Chen, Joyce Chuyi Dormoy, Ines Kim, Jinyoung Khanna, Samar Zheng, Zhuo Ermon, Stefano |
| author_facet | Irvin, Jeremy Andrew Liu, Emily Ruoyu Chen, Joyce Chuyi Dormoy, Ines Kim, Jinyoung Khanna, Samar Zheng, Zhuo Ermon, Stefano |
| contents | Large vision and language assistants have enabled new capabilities for interpreting natural images. These approaches have recently been adapted to earth observation data, but they are only able to handle single image inputs, limiting their use for many real-world tasks. In this work, we develop a new vision and language assistant called TEOChat that can engage in conversations about temporal sequences of earth observation data. To train TEOChat, we curate an instruction-following dataset composed of many single image and temporal tasks including building change and damage assessment, semantic change detection, and temporal scene classification. We show that TEOChat can perform a wide variety of spatial and temporal reasoning tasks, substantially outperforming previous vision and language assistants, and even achieving comparable or better performance than several specialist models trained to perform specific tasks. Furthermore, TEOChat achieves impressive zero-shot performance on a change detection and change question answering dataset, outperforms GPT-4o and Gemini 1.5 Pro on multiple temporal tasks, and exhibits stronger single image capabilities than a comparable single image instruction-following model on scene classification, visual question answering, and captioning. We publicly release our data, model, and code at https://github.com/ermongroup/TEOChat . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_06234 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data Irvin, Jeremy Andrew Liu, Emily Ruoyu Chen, Joyce Chuyi Dormoy, Ines Kim, Jinyoung Khanna, Samar Zheng, Zhuo Ermon, Stefano Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Large vision and language assistants have enabled new capabilities for interpreting natural images. These approaches have recently been adapted to earth observation data, but they are only able to handle single image inputs, limiting their use for many real-world tasks. In this work, we develop a new vision and language assistant called TEOChat that can engage in conversations about temporal sequences of earth observation data. To train TEOChat, we curate an instruction-following dataset composed of many single image and temporal tasks including building change and damage assessment, semantic change detection, and temporal scene classification. We show that TEOChat can perform a wide variety of spatial and temporal reasoning tasks, substantially outperforming previous vision and language assistants, and even achieving comparable or better performance than several specialist models trained to perform specific tasks. Furthermore, TEOChat achieves impressive zero-shot performance on a change detection and change question answering dataset, outperforms GPT-4o and Gemini 1.5 Pro on multiple temporal tasks, and exhibits stronger single image capabilities than a comparable single image instruction-following model on scene classification, visual question answering, and captioning. We publicly release our data, model, and code at https://github.com/ermongroup/TEOChat . |
| title | TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2410.06234 |