HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916668774547456 |
|---|---|
| author | Wang, Di Hu, Meiqi Jin, Yao Miao, Yuchun Yang, Jiaqi Xu, Yichu Qin, Xiaolei Ma, Jiaqi Sun, Lingyu Li, Chenxing Fu, Chuan Chen, Hongruixuan Han, Chengxi Yokoya, Naoto Zhang, Jing Xu, Minqiang Liu, Lin Zhang, Lefei Wu, Chen Du, Bo Tao, Dacheng Zhang, Liangpei |
| author_facet | Wang, Di Hu, Meiqi Jin, Yao Miao, Yuchun Yang, Jiaqi Xu, Yichu Qin, Xiaolei Ma, Jiaqi Sun, Lingyu Li, Chenxing Fu, Chuan Chen, Hongruixuan Han, Chengxi Yokoya, Naoto Zhang, Jing Xu, Minqiang Liu, Lin Zhang, Lefei Wu, Chen Du, Bo Tao, Dacheng Zhang, Liangpei |
| contents | Accurate hyperspectral image (HSI) interpretation is critical for providing valuable insights into various earth observation-related applications such as urban planning, precision agriculture, and environmental monitoring. However, existing HSI processing methods are predominantly task-specific and scene-dependent, which severely limits their ability to transfer knowledge across tasks and scenes, thereby reducing the practicality in real-world applications. To address these challenges, we present HyperSIGMA, a vision transformer-based foundation model that unifies HSI interpretation across tasks and scenes, scalable to over one billion parameters. To overcome the spectral and spatial redundancy inherent in HSIs, we introduce a novel sparse sampling attention (SSA) mechanism, which effectively promotes the learning of diverse contextual features and serves as the basic block of HyperSIGMA. HyperSIGMA integrates spatial and spectral features using a specially designed spectral enhancement module. In addition, we construct a large-scale hyperspectral dataset, HyperGlobal-450K, for pre-training, which contains about 450K hyperspectral images, significantly surpassing existing datasets in scale. Extensive experiments on various high-level and low-level HSI tasks demonstrate HyperSIGMA's versatility and superior representational capability compared to current state-of-the-art methods. Moreover, HyperSIGMA shows significant advantages in scalability, robustness, cross-modal transferring capability, real-world applicability, and computational efficiency. The code and models will be released at https://github.com/WHU-Sigma/HyperSIGMA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_11519 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Model Wang, Di Hu, Meiqi Jin, Yao Miao, Yuchun Yang, Jiaqi Xu, Yichu Qin, Xiaolei Ma, Jiaqi Sun, Lingyu Li, Chenxing Fu, Chuan Chen, Hongruixuan Han, Chengxi Yokoya, Naoto Zhang, Jing Xu, Minqiang Liu, Lin Zhang, Lefei Wu, Chen Du, Bo Tao, Dacheng Zhang, Liangpei Computer Vision and Pattern Recognition Image and Video Processing Accurate hyperspectral image (HSI) interpretation is critical for providing valuable insights into various earth observation-related applications such as urban planning, precision agriculture, and environmental monitoring. However, existing HSI processing methods are predominantly task-specific and scene-dependent, which severely limits their ability to transfer knowledge across tasks and scenes, thereby reducing the practicality in real-world applications. To address these challenges, we present HyperSIGMA, a vision transformer-based foundation model that unifies HSI interpretation across tasks and scenes, scalable to over one billion parameters. To overcome the spectral and spatial redundancy inherent in HSIs, we introduce a novel sparse sampling attention (SSA) mechanism, which effectively promotes the learning of diverse contextual features and serves as the basic block of HyperSIGMA. HyperSIGMA integrates spatial and spectral features using a specially designed spectral enhancement module. In addition, we construct a large-scale hyperspectral dataset, HyperGlobal-450K, for pre-training, which contains about 450K hyperspectral images, significantly surpassing existing datasets in scale. Extensive experiments on various high-level and low-level HSI tasks demonstrate HyperSIGMA's versatility and superior representational capability compared to current state-of-the-art methods. Moreover, HyperSIGMA shows significant advantages in scalability, robustness, cross-modal transferring capability, real-world applicability, and computational efficiency. The code and models will be released at https://github.com/WHU-Sigma/HyperSIGMA. |
| title | HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Model |
| topic | Computer Vision and Pattern Recognition Image and Video Processing |
| url | https://arxiv.org/abs/2406.11519 |