Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.09939 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916318088790016 |
|---|---|
| author | Wan, Yuwei Liu, Yixuan Ajith, Aswathy Grazian, Clara Hoex, Bram Zhang, Wenjie Kit, Chunyu Xie, Tong Foster, Ian |
| author_facet | Wan, Yuwei Liu, Yixuan Ajith, Aswathy Grazian, Clara Hoex, Bram Zhang, Wenjie Kit, Chunyu Xie, Tong Foster, Ian |
| contents | We introduce SciQAG, a novel framework for automatically generating high-quality science question-answer pairs from a large corpus of scientific literature based on large language models (LLMs). SciQAG consists of a QA generator and a QA evaluator, which work together to extract diverse and research-level questions and answers from scientific papers. Utilizing this framework, we construct a large-scale, high-quality, open-ended science QA dataset containing 188,042 QA pairs extracted from 22,743 scientific papers across 24 scientific domains. We also introduce SciQAG-24D, a new benchmark task designed to evaluate the science question-answering ability of LLMs. Extensive experiments demonstrate that fine-tuning LLMs on the SciQAG dataset significantly improves their performance on both open-ended question answering and scientific tasks. To foster research and collaboration, we make the datasets, models, and evaluation codes publicly available, contributing to the advancement of science question answering and developing more interpretable and reasoning-capable AI systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_09939 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluation Wan, Yuwei Liu, Yixuan Ajith, Aswathy Grazian, Clara Hoex, Bram Zhang, Wenjie Kit, Chunyu Xie, Tong Foster, Ian Computation and Language Artificial Intelligence We introduce SciQAG, a novel framework for automatically generating high-quality science question-answer pairs from a large corpus of scientific literature based on large language models (LLMs). SciQAG consists of a QA generator and a QA evaluator, which work together to extract diverse and research-level questions and answers from scientific papers. Utilizing this framework, we construct a large-scale, high-quality, open-ended science QA dataset containing 188,042 QA pairs extracted from 22,743 scientific papers across 24 scientific domains. We also introduce SciQAG-24D, a new benchmark task designed to evaluate the science question-answering ability of LLMs. Extensive experiments demonstrate that fine-tuning LLMs on the SciQAG dataset significantly improves their performance on both open-ended question answering and scientific tasks. To foster research and collaboration, we make the datasets, models, and evaluation codes publicly available, contributing to the advancement of science question answering and developing more interpretable and reasoning-capable AI systems. |
| title | SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2405.09939 |