AstroLLaVA: towards the unification of astronomical data and natural language
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917982644469760 |
|---|---|
| author | Zaman, Sharaf Smith, Michael J. Khetarpal, Pranav Chakrabarty, Rishabh Ginolfi, Michele Huertas-Company, Marc Jabłońska, Maja Kruk, Sandor Lain, Matthieu Le Méndez, Sergio José Rodríguez Tanoglidis, Dimitrios |
| author_facet | Zaman, Sharaf Smith, Michael J. Khetarpal, Pranav Chakrabarty, Rishabh Ginolfi, Michele Huertas-Company, Marc Jabłońska, Maja Kruk, Sandor Lain, Matthieu Le Méndez, Sergio José Rodríguez Tanoglidis, Dimitrios |
| contents | We present AstroLLaVA, a vision language model for astronomy that enables interaction with astronomical imagery through natural dialogue. By fine-tuning the LLaVA model on a diverse dataset of $\sim$30k images with captions and question-answer pairs sourced from NASA's `Astronomy Picture of the Day', the European Southern Observatory, and the NASA/ESA Hubble Space Telescope, we create a model capable of answering open-ended questions about astronomical concepts depicted visually. Our two-stage fine-tuning process adapts the model to both image captioning and visual question answering in the astronomy domain. We demonstrate AstroLLaVA's performance on an astronomical visual question answering benchmark and release the model weights, code, and training set to encourage further open source work in this space. Finally, we suggest a roadmap towards general astronomical data alignment with pre-trained language models, and provide an open space for collaboration towards this end for interested researchers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_08583 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AstroLLaVA: towards the unification of astronomical data and natural language Zaman, Sharaf Smith, Michael J. Khetarpal, Pranav Chakrabarty, Rishabh Ginolfi, Michele Huertas-Company, Marc Jabłońska, Maja Kruk, Sandor Lain, Matthieu Le Méndez, Sergio José Rodríguez Tanoglidis, Dimitrios Instrumentation and Methods for Astrophysics Machine Learning We present AstroLLaVA, a vision language model for astronomy that enables interaction with astronomical imagery through natural dialogue. By fine-tuning the LLaVA model on a diverse dataset of $\sim$30k images with captions and question-answer pairs sourced from NASA's `Astronomy Picture of the Day', the European Southern Observatory, and the NASA/ESA Hubble Space Telescope, we create a model capable of answering open-ended questions about astronomical concepts depicted visually. Our two-stage fine-tuning process adapts the model to both image captioning and visual question answering in the astronomy domain. We demonstrate AstroLLaVA's performance on an astronomical visual question answering benchmark and release the model weights, code, and training set to encourage further open source work in this space. Finally, we suggest a roadmap towards general astronomical data alignment with pre-trained language models, and provide an open space for collaboration towards this end for interested researchers. |
| title | AstroLLaVA: towards the unification of astronomical data and natural language |
| topic | Instrumentation and Methods for Astrophysics Machine Learning |
| url | https://arxiv.org/abs/2504.08583 |