OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914992168632320 |
|---|---|
| author | Dias, Philipe Tsaris, Aristeidis Bowman, Jordan Potnis, Abhishek Arndt, Jacob Yang, H. Lexie Lunga, Dalton |
| author_facet | Dias, Philipe Tsaris, Aristeidis Bowman, Jordan Potnis, Abhishek Arndt, Jacob Yang, H. Lexie Lunga, Dalton |
| contents | While the pretraining of Foundation Models (FMs) for remote sensing (RS) imagery is on the rise, models remain restricted to a few hundred million parameters. Scaling models to billions of parameters has been shown to yield unprecedented benefits including emergent abilities, but requires data scaling and computing resources typically not available outside industry R&D labs. In this work, we pair high-performance computing resources including Frontier supercomputer, America's first exascale system, and high-resolution optical RS data to pretrain billion-scale FMs. Our study assesses performance of different pretrained variants of vision Transformers across image classification, semantic segmentation and object detection benchmarks, which highlight the importance of data scaling for effective model scaling. Moreover, we discuss construction of a novel TIU pretraining dataset, model initialization, with data and pretrained models intended for public release. By discussing technical challenges and details often lacking in the related literature, this work is intended to offer best practices to the geospatial community toward efficient training and benchmarking of larger FMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_19965 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery Dias, Philipe Tsaris, Aristeidis Bowman, Jordan Potnis, Abhishek Arndt, Jacob Yang, H. Lexie Lunga, Dalton Computer Vision and Pattern Recognition Artificial Intelligence Distributed, Parallel, and Cluster Computing While the pretraining of Foundation Models (FMs) for remote sensing (RS) imagery is on the rise, models remain restricted to a few hundred million parameters. Scaling models to billions of parameters has been shown to yield unprecedented benefits including emergent abilities, but requires data scaling and computing resources typically not available outside industry R&D labs. In this work, we pair high-performance computing resources including Frontier supercomputer, America's first exascale system, and high-resolution optical RS data to pretrain billion-scale FMs. Our study assesses performance of different pretrained variants of vision Transformers across image classification, semantic segmentation and object detection benchmarks, which highlight the importance of data scaling for effective model scaling. Moreover, we discuss construction of a novel TIU pretraining dataset, model initialization, with data and pretrained models intended for public release. By discussing technical challenges and details often lacking in the related literature, this work is intended to offer best practices to the geospatial community toward efficient training and benchmarking of larger FMs. |
| title | OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2410.19965 |