GRS-QA -- Graph Reasoning-Structured Question Answering Dataset
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916472742215680 |
|---|---|
| author | Pahilajani, Anish Trivedi, Devasha Shuai, Jincen Yone, Khin S. Jain, Samyak Rajesh Park, Namyong Rossi, Ryan A. Ahmed, Nesreen K. Dernoncourt, Franck Wang, Yu |
| author_facet | Pahilajani, Anish Trivedi, Devasha Shuai, Jincen Yone, Khin S. Jain, Samyak Rajesh Park, Namyong Rossi, Ryan A. Ahmed, Nesreen K. Dernoncourt, Franck Wang, Yu |
| contents | Large Language Models (LLMs) have excelled in multi-hop question-answering (M-QA) due to their advanced reasoning abilities. However, the impact of the inherent reasoning structures on LLM M-QA performance remains unclear, largely due to the absence of QA datasets that provide fine-grained reasoning structures. To address this gap, we introduce the Graph Reasoning-Structured Question Answering Dataset (GRS-QA), which includes both semantic contexts and reasoning structures for QA pairs. Unlike existing M-QA datasets, where different reasoning structures are entangled together, GRS-QA explicitly captures intricate reasoning pathways by constructing reasoning graphs, where nodes represent textual contexts and edges denote logical flows. These reasoning graphs of different structures enable a fine-grained evaluation of LLM reasoning capabilities across various reasoning structures. Our empirical analysis reveals that LLMs perform differently when handling questions with varying reasoning structures. This finding facilitates the exploration of textual structures as compared with semantics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_00369 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GRS-QA -- Graph Reasoning-Structured Question Answering Dataset Pahilajani, Anish Trivedi, Devasha Shuai, Jincen Yone, Khin S. Jain, Samyak Rajesh Park, Namyong Rossi, Ryan A. Ahmed, Nesreen K. Dernoncourt, Franck Wang, Yu Computation and Language Large Language Models (LLMs) have excelled in multi-hop question-answering (M-QA) due to their advanced reasoning abilities. However, the impact of the inherent reasoning structures on LLM M-QA performance remains unclear, largely due to the absence of QA datasets that provide fine-grained reasoning structures. To address this gap, we introduce the Graph Reasoning-Structured Question Answering Dataset (GRS-QA), which includes both semantic contexts and reasoning structures for QA pairs. Unlike existing M-QA datasets, where different reasoning structures are entangled together, GRS-QA explicitly captures intricate reasoning pathways by constructing reasoning graphs, where nodes represent textual contexts and edges denote logical flows. These reasoning graphs of different structures enable a fine-grained evaluation of LLM reasoning capabilities across various reasoning structures. Our empirical analysis reveals that LLMs perform differently when handling questions with varying reasoning structures. This finding facilitates the exploration of textual structures as compared with semantics. |
| title | GRS-QA -- Graph Reasoning-Structured Question Answering Dataset |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.00369 |