Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.05714 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916473995264000 |
|---|---|
| author | King, Ethan Rodriguez, Jaime Llanes, Diego Doster, Timothy Emerson, Tegan Koch, James |
| author_facet | King, Ethan Rodriguez, Jaime Llanes, Diego Doster, Timothy Emerson, Tegan Koch, James |
| contents | We present a sensor-agnostic spectral transformer as the basis for spectral foundation models. To that end, we introduce a Universal Spectral Representation (USR) that leverages sensor meta-data, such as sensing kernel specifications and sensing wavelengths, to encode spectra obtained from any spectral instrument into a common representation, such that a single model can ingest data from any sensor. Furthermore, we develop a methodology for pre-training such models in a self-supervised manner using a novel random sensor-augmentation and reconstruction pipeline to learn spectral features independent of the sensing paradigm. We demonstrate that our architecture can learn sensor independent spectral features that generalize effectively to sensors not seen during training. This work sets the stage for training foundation models that can both leverage and be effective for the growing diversity of spectral data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_05714 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | STARS: Sensor-agnostic Transformer Architecture for Remote Sensing King, Ethan Rodriguez, Jaime Llanes, Diego Doster, Timothy Emerson, Tegan Koch, James Computer Vision and Pattern Recognition Machine Learning Image and Video Processing We present a sensor-agnostic spectral transformer as the basis for spectral foundation models. To that end, we introduce a Universal Spectral Representation (USR) that leverages sensor meta-data, such as sensing kernel specifications and sensing wavelengths, to encode spectra obtained from any spectral instrument into a common representation, such that a single model can ingest data from any sensor. Furthermore, we develop a methodology for pre-training such models in a self-supervised manner using a novel random sensor-augmentation and reconstruction pipeline to learn spectral features independent of the sensing paradigm. We demonstrate that our architecture can learn sensor independent spectral features that generalize effectively to sensors not seen during training. This work sets the stage for training foundation models that can both leverage and be effective for the growing diversity of spectral data. |
| title | STARS: Sensor-agnostic Transformer Architecture for Remote Sensing |
| topic | Computer Vision and Pattern Recognition Machine Learning Image and Video Processing |
| url | https://arxiv.org/abs/2411.05714 |