SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916973700448256 |
|---|---|
| author | Wadden, David Shi, Kejian Morrison, Jacob Li, Alan Naik, Aakanksha Singh, Shruti Barzilay, Nitzan Lo, Kyle Hope, Tom Soldaini, Luca Shen, Shannon Zejiang Downey, Doug Hajishirzi, Hannaneh Cohan, Arman |
| author_facet | Wadden, David Shi, Kejian Morrison, Jacob Li, Alan Naik, Aakanksha Singh, Shruti Barzilay, Nitzan Lo, Kyle Hope, Tom Soldaini, Luca Shen, Shannon Zejiang Downey, Doug Hajishirzi, Hannaneh Cohan, Arman |
| contents | We present SciRIFF (Scientific Resource for Instruction-Following and Finetuning), a dataset of 137K instruction-following instances for training and evaluation, covering 54 tasks. These tasks span five core scientific literature understanding capabilities: information extraction, summarization, question answering, claim verification, and classification. SciRIFF is unique in being entirely expert-written, high-quality instruction-following dataset for extracting and synthesizing information from research literature across diverse scientific fields. It features complex instructions with long input contexts, detailed task descriptions, and structured outputs. To demonstrate its utility, we finetune a series of large language models (LLMs) using a mix of general-domain and SciRIFF instructions. On nine out-of-distribution held-out tasks (referred to as SciRIFF-Eval), LLMs finetuned on SciRIFF achieve 70.6% average improvement over baselines trained only on general-domain instructions. SciRIFF facilitates the development and evaluation of LLMs to help researchers navigate the rapidly growing body of scientific literature. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07835 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature Wadden, David Shi, Kejian Morrison, Jacob Li, Alan Naik, Aakanksha Singh, Shruti Barzilay, Nitzan Lo, Kyle Hope, Tom Soldaini, Luca Shen, Shannon Zejiang Downey, Doug Hajishirzi, Hannaneh Cohan, Arman Computation and Language Artificial Intelligence We present SciRIFF (Scientific Resource for Instruction-Following and Finetuning), a dataset of 137K instruction-following instances for training and evaluation, covering 54 tasks. These tasks span five core scientific literature understanding capabilities: information extraction, summarization, question answering, claim verification, and classification. SciRIFF is unique in being entirely expert-written, high-quality instruction-following dataset for extracting and synthesizing information from research literature across diverse scientific fields. It features complex instructions with long input contexts, detailed task descriptions, and structured outputs. To demonstrate its utility, we finetune a series of large language models (LLMs) using a mix of general-domain and SciRIFF instructions. On nine out-of-distribution held-out tasks (referred to as SciRIFF-Eval), LLMs finetuned on SciRIFF achieve 70.6% average improvement over baselines trained only on general-domain instructions. SciRIFF facilitates the development and evaluation of LLMs to help researchers navigate the rapidly growing body of scientific literature. |
| title | SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2406.07835 |