Designing an Evaluation Framework for Large Language Models in Astronomy Research
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913371372126208 |
|---|---|
| author | Wu, John F. Hyk, Alina McCormick, Kiera Ye, Christine Astarita, Simone Baral, Elina Ciuca, Jo Cranney, Jesse Field, Anjalie Iyer, Kartheik Koehn, Philipp Kotler, Jenn Kruk, Sandor Ntampaka, Michelle O'Neill, Charles Peek, Joshua E. G. Sharma, Sanjib Yunus, Mikaeel |
| author_facet | Wu, John F. Hyk, Alina McCormick, Kiera Ye, Christine Astarita, Simone Baral, Elina Ciuca, Jo Cranney, Jesse Field, Anjalie Iyer, Kartheik Koehn, Philipp Kotler, Jenn Kruk, Sandor Ntampaka, Michelle O'Neill, Charles Peek, Joshua E. G. Sharma, Sanjib Yunus, Mikaeel |
| contents | Large Language Models (LLMs) are shifting how scientific research is done. It is imperative to understand how researchers interact with these models and how scientific sub-communities like astronomy might benefit from them. However, there is currently no standard for evaluating the use of LLMs in astronomy. Therefore, we present the experimental design for an evaluation study on how astronomy researchers interact with LLMs. We deploy a Slack chatbot that can answer queries from users via Retrieval-Augmented Generation (RAG); these responses are grounded in astronomy papers from arXiv. We record and anonymize user questions and chatbot answers, user upvotes and downvotes to LLM responses, user feedback to the LLM, and retrieved documents and similarity scores with the query. Our data collection method will enable future dynamic evaluations of LLM tools for astronomy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_20389 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Designing an Evaluation Framework for Large Language Models in Astronomy Research Wu, John F. Hyk, Alina McCormick, Kiera Ye, Christine Astarita, Simone Baral, Elina Ciuca, Jo Cranney, Jesse Field, Anjalie Iyer, Kartheik Koehn, Philipp Kotler, Jenn Kruk, Sandor Ntampaka, Michelle O'Neill, Charles Peek, Joshua E. G. Sharma, Sanjib Yunus, Mikaeel Instrumentation and Methods for Astrophysics Artificial Intelligence Human-Computer Interaction Information Retrieval Large Language Models (LLMs) are shifting how scientific research is done. It is imperative to understand how researchers interact with these models and how scientific sub-communities like astronomy might benefit from them. However, there is currently no standard for evaluating the use of LLMs in astronomy. Therefore, we present the experimental design for an evaluation study on how astronomy researchers interact with LLMs. We deploy a Slack chatbot that can answer queries from users via Retrieval-Augmented Generation (RAG); these responses are grounded in astronomy papers from arXiv. We record and anonymize user questions and chatbot answers, user upvotes and downvotes to LLM responses, user feedback to the LLM, and retrieved documents and similarity scores with the query. Our data collection method will enable future dynamic evaluations of LLM tools for astronomy. |
| title | Designing an Evaluation Framework for Large Language Models in Astronomy Research |
| topic | Instrumentation and Methods for Astrophysics Artificial Intelligence Human-Computer Interaction Information Retrieval |
| url | https://arxiv.org/abs/2405.20389 |