AstroCLIP: A Cross-Modal Foundation Model for Galaxies
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916286503583744 |
|---|---|
| author | Parker, Liam Lanusse, Francois Golkar, Siavash Sarra, Leopoldo Cranmer, Miles Bietti, Alberto Eickenberg, Michael Krawezik, Geraud McCabe, Michael Ohana, Ruben Pettee, Mariel Blancard, Bruno Regaldo-Saint Tesileanu, Tiberiu Cho, Kyunghyun Ho, Shirley |
| author_facet | Parker, Liam Lanusse, Francois Golkar, Siavash Sarra, Leopoldo Cranmer, Miles Bietti, Alberto Eickenberg, Michael Krawezik, Geraud McCabe, Michael Ohana, Ruben Pettee, Mariel Blancard, Bruno Regaldo-Saint Tesileanu, Tiberiu Cho, Kyunghyun Ho, Shirley |
| contents | We present AstroCLIP, a single, versatile model that can embed both galaxy images and spectra into a shared, physically meaningful latent space. These embeddings can then be used - without any model fine-tuning - for a variety of downstream tasks including (1) accurate in-modality and cross-modality semantic similarity search, (2) photometric redshift estimation, (3) galaxy property estimation from both images and spectra, and (4) morphology classification. Our approach to implementing AstroCLIP consists of two parts. First, we embed galaxy images and spectra separately by pretraining separate transformer-based image and spectrum encoders in self-supervised settings. We then align the encoders using a contrastive loss. We apply our method to spectra from the Dark Energy Spectroscopic Instrument and images from its corresponding Legacy Imaging Survey. Overall, we find remarkable performance on all downstream tasks, even relative to supervised baselines. For example, for a task like photometric redshift prediction, we find similar performance to a specifically-trained ResNet18, and for additional tasks like physical property estimation (stellar mass, age, metallicity, and sSFR), we beat this supervised baseline by 19\% in terms of $R^2$. We also compare our results to a state-of-the-art self-supervised single-modal model for galaxy images, and find that our approach outperforms this benchmark by roughly a factor of two on photometric redshift estimation and physical property prediction in terms of $R^2$, while remaining roughly in-line in terms of morphology classification. Ultimately, our approach represents the first cross-modal self-supervised model for galaxies, and the first self-supervised transformer-based architectures for galaxy images and spectra. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_03024 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | AstroCLIP: A Cross-Modal Foundation Model for Galaxies Parker, Liam Lanusse, Francois Golkar, Siavash Sarra, Leopoldo Cranmer, Miles Bietti, Alberto Eickenberg, Michael Krawezik, Geraud McCabe, Michael Ohana, Ruben Pettee, Mariel Blancard, Bruno Regaldo-Saint Tesileanu, Tiberiu Cho, Kyunghyun Ho, Shirley Instrumentation and Methods for Astrophysics Artificial Intelligence Machine Learning We present AstroCLIP, a single, versatile model that can embed both galaxy images and spectra into a shared, physically meaningful latent space. These embeddings can then be used - without any model fine-tuning - for a variety of downstream tasks including (1) accurate in-modality and cross-modality semantic similarity search, (2) photometric redshift estimation, (3) galaxy property estimation from both images and spectra, and (4) morphology classification. Our approach to implementing AstroCLIP consists of two parts. First, we embed galaxy images and spectra separately by pretraining separate transformer-based image and spectrum encoders in self-supervised settings. We then align the encoders using a contrastive loss. We apply our method to spectra from the Dark Energy Spectroscopic Instrument and images from its corresponding Legacy Imaging Survey. Overall, we find remarkable performance on all downstream tasks, even relative to supervised baselines. For example, for a task like photometric redshift prediction, we find similar performance to a specifically-trained ResNet18, and for additional tasks like physical property estimation (stellar mass, age, metallicity, and sSFR), we beat this supervised baseline by 19\% in terms of $R^2$. We also compare our results to a state-of-the-art self-supervised single-modal model for galaxy images, and find that our approach outperforms this benchmark by roughly a factor of two on photometric redshift estimation and physical property prediction in terms of $R^2$, while remaining roughly in-line in terms of morphology classification. Ultimately, our approach represents the first cross-modal self-supervised model for galaxies, and the first self-supervised transformer-based architectures for galaxy images and spectra. |
| title | AstroCLIP: A Cross-Modal Foundation Model for Galaxies |
| topic | Instrumentation and Methods for Astrophysics Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2310.03024 |