AstroMLab 4: Benchmark-Topping Performance in Astronomy Q&A with a 70B-Parameter Domain-Specialized Reasoning Model
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914339295854592 |
|---|---|
| author | de Haan, Tijmen Ting, Yuan-Sen Ghosal, Tirthankar Nguyen, Tuan Dung Accomazzi, Alberto Herron, Emily Lama, Vanessa Pan, Rui Wells, Azton Ramachandra, Nesar |
| author_facet | de Haan, Tijmen Ting, Yuan-Sen Ghosal, Tirthankar Nguyen, Tuan Dung Accomazzi, Alberto Herron, Emily Lama, Vanessa Pan, Rui Wells, Azton Ramachandra, Nesar |
| contents | General-purpose large language models (LLMs), despite their broad capabilities, often struggle with specialized domain knowledge. This gap hinders their deployment as reliable research agents in demanding fields such as astronomy. Building on our prior work with AstroSage-Llama-3.1-8B, this study introduces AstroSage-Llama-3.1-70B, a 70-billion parameter domain-specialized natural-language AI assistant. It is designed for research and education across astronomy, astrophysics, space science, astroparticle physics, cosmology, and astronomical instrumentation. Developed from the Meta-Llama-3.1-70B foundation, AstroSage-Llama-3.1-70B underwent extensive continued pre-training (CPT) on a vast corpus of astronomical literature, followed by supervised fine-tuning (SFT) and model merging. We integrated reasoning chains into the SFT dataset, enabling AstroSage-Llama-3.1-70B to either answer the user query immediately, or first emit a human-readable thought process. Evaluated on a validated subset of 3,846 questions from the AstroMLab-1 benchmark (Ting et al., 2024) -- derived from literature withheld during training -- AstroSage-Llama-3.1-70B achieves top-tier performance (89.0%), matching GPT-5.2, Claude-4.5-Opus, and Gemini-3-Pro while being more cost-efficient. This work demonstrates that domain specialization, when applied to large-scale models, can enable them to outperform generalist counterparts in specialized knowledge areas like astronomy, thereby advancing the frontier of AI capabilities in the field. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_17592 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AstroMLab 4: Benchmark-Topping Performance in Astronomy Q&A with a 70B-Parameter Domain-Specialized Reasoning Model de Haan, Tijmen Ting, Yuan-Sen Ghosal, Tirthankar Nguyen, Tuan Dung Accomazzi, Alberto Herron, Emily Lama, Vanessa Pan, Rui Wells, Azton Ramachandra, Nesar Instrumentation and Methods for Astrophysics Machine Learning General-purpose large language models (LLMs), despite their broad capabilities, often struggle with specialized domain knowledge. This gap hinders their deployment as reliable research agents in demanding fields such as astronomy. Building on our prior work with AstroSage-Llama-3.1-8B, this study introduces AstroSage-Llama-3.1-70B, a 70-billion parameter domain-specialized natural-language AI assistant. It is designed for research and education across astronomy, astrophysics, space science, astroparticle physics, cosmology, and astronomical instrumentation. Developed from the Meta-Llama-3.1-70B foundation, AstroSage-Llama-3.1-70B underwent extensive continued pre-training (CPT) on a vast corpus of astronomical literature, followed by supervised fine-tuning (SFT) and model merging. We integrated reasoning chains into the SFT dataset, enabling AstroSage-Llama-3.1-70B to either answer the user query immediately, or first emit a human-readable thought process. Evaluated on a validated subset of 3,846 questions from the AstroMLab-1 benchmark (Ting et al., 2024) -- derived from literature withheld during training -- AstroSage-Llama-3.1-70B achieves top-tier performance (89.0%), matching GPT-5.2, Claude-4.5-Opus, and Gemini-3-Pro while being more cost-efficient. This work demonstrates that domain specialization, when applied to large-scale models, can enable them to outperform generalist counterparts in specialized knowledge areas like astronomy, thereby advancing the frontier of AI capabilities in the field. |
| title | AstroMLab 4: Benchmark-Topping Performance in Astronomy Q&A with a 70B-Parameter Domain-Specialized Reasoning Model |
| topic | Instrumentation and Methods for Astrophysics Machine Learning |
| url | https://arxiv.org/abs/2505.17592 |