ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910866174115840 |
|---|---|
| author | Manivannan, Veeramakali Vignesh Jafari, Yasaman Eranky, Srikar Ho, Spencer Yu, Rose Watson-Parris, Duncan Ma, Yian Bergen, Leon Berg-Kirkpatrick, Taylor |
| author_facet | Manivannan, Veeramakali Vignesh Jafari, Yasaman Eranky, Srikar Ho, Spencer Yu, Rose Watson-Parris, Duncan Ma, Yian Bergen, Leon Berg-Kirkpatrick, Taylor |
| contents | The use of Large Language Models (LLMs) in climate science has recently gained significant attention. However, a critical issue remains: the lack of a comprehensive evaluation framework capable of assessing the quality and scientific validity of model outputs. To address this issue, we develop ClimaGen (Climate QA Generator), an adaptive learning framework that generates question-answer pairs from graduate textbooks with climate scientists in the loop. As a result, we present ClimaQA-Gold, an expert-annotated benchmark dataset alongside ClimaQA-Silver, a large-scale, comprehensive synthetic QA dataset for climate science. Finally, we develop evaluation strategies and compare different LLMs on our benchmarks. Our results offer novel insights into various approaches used to enhance knowledge of climate LLMs. The source code is publicly available at https://github.com/Rose-STL-Lab/genie-climaqa |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_16701 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models Manivannan, Veeramakali Vignesh Jafari, Yasaman Eranky, Srikar Ho, Spencer Yu, Rose Watson-Parris, Duncan Ma, Yian Bergen, Leon Berg-Kirkpatrick, Taylor Machine Learning The use of Large Language Models (LLMs) in climate science has recently gained significant attention. However, a critical issue remains: the lack of a comprehensive evaluation framework capable of assessing the quality and scientific validity of model outputs. To address this issue, we develop ClimaGen (Climate QA Generator), an adaptive learning framework that generates question-answer pairs from graduate textbooks with climate scientists in the loop. As a result, we present ClimaQA-Gold, an expert-annotated benchmark dataset alongside ClimaQA-Silver, a large-scale, comprehensive synthetic QA dataset for climate science. Finally, we develop evaluation strategies and compare different LLMs on our benchmarks. Our results offer novel insights into various approaches used to enhance knowledge of climate LLMs. The source code is publicly available at https://github.com/Rose-STL-Lab/genie-climaqa |
| title | ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.16701 |