AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929199183298560 |
|---|---|
| author | Perkowski, Ernest Pan, Rui Nguyen, Tuan Dung Ting, Yuan-Sen Kruk, Sandor Zhang, Tong O'Neill, Charlie Jablonska, Maja Sun, Zechang Smith, Michael J. Liu, Huiling Schawinski, Kevin Iyer, Kartheik UniverseTBD, Ioana Ciucă for |
| author_facet | Perkowski, Ernest Pan, Rui Nguyen, Tuan Dung Ting, Yuan-Sen Kruk, Sandor Zhang, Tong O'Neill, Charlie Jablonska, Maja Sun, Zechang Smith, Michael J. Liu, Huiling Schawinski, Kevin Iyer, Kartheik UniverseTBD, Ioana Ciucă for |
| contents | We explore the potential of enhancing LLM performance in astronomy-focused question-answering through targeted, continual pre-training. By employing a compact 7B-parameter LLaMA-2 model and focusing exclusively on a curated set of astronomy corpora -- comprising abstracts, introductions, and conclusions -- we achieve notable improvements in specialized topic comprehension. While general LLMs like GPT-4 excel in broader question-answering scenarios due to superior reasoning capabilities, our findings suggest that continual pre-training with limited resources can still enhance model performance on specialized topics. Additionally, we present an extension of AstroLLaMA: the fine-tuning of the 7B LLaMA model on a domain-specific conversational dataset, culminating in the release of the chat-enabled AstroLLaMA for community use. Comprehensive quantitative benchmarking is currently in progress and will be detailed in an upcoming full paper. The model, AstroLLaMA-Chat, is now available at https://huggingface.co/universeTBD, providing the first open-source conversational AI tool tailored for the astronomy community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_01916 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets Perkowski, Ernest Pan, Rui Nguyen, Tuan Dung Ting, Yuan-Sen Kruk, Sandor Zhang, Tong O'Neill, Charlie Jablonska, Maja Sun, Zechang Smith, Michael J. Liu, Huiling Schawinski, Kevin Iyer, Kartheik UniverseTBD, Ioana Ciucă for Instrumentation and Methods for Astrophysics Cosmology and Nongalactic Astrophysics Astrophysics of Galaxies Solar and Stellar Astrophysics Computation and Language Machine Learning We explore the potential of enhancing LLM performance in astronomy-focused question-answering through targeted, continual pre-training. By employing a compact 7B-parameter LLaMA-2 model and focusing exclusively on a curated set of astronomy corpora -- comprising abstracts, introductions, and conclusions -- we achieve notable improvements in specialized topic comprehension. While general LLMs like GPT-4 excel in broader question-answering scenarios due to superior reasoning capabilities, our findings suggest that continual pre-training with limited resources can still enhance model performance on specialized topics. Additionally, we present an extension of AstroLLaMA: the fine-tuning of the 7B LLaMA model on a domain-specific conversational dataset, culminating in the release of the chat-enabled AstroLLaMA for community use. Comprehensive quantitative benchmarking is currently in progress and will be detailed in an upcoming full paper. The model, AstroLLaMA-Chat, is now available at https://huggingface.co/universeTBD, providing the first open-source conversational AI tool tailored for the astronomy community. |
| title | AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets |
| topic | Instrumentation and Methods for Astrophysics Cosmology and Nongalactic Astrophysics Astrophysics of Galaxies Solar and Stellar Astrophysics Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2401.01916 |