DisastQA: A Comprehensive Benchmark for Evaluating Question Answering in Disaster Management
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909983319261184 |
|---|---|
| author | Chen, Zhitong Yin, Kai Dong, Xiangjue Liu, Chengkai Li, Xiangpeng Xiao, Yiming Li, Bo Ma, Junwei Mostafavi, Ali Caverlee, James |
| author_facet | Chen, Zhitong Yin, Kai Dong, Xiangjue Liu, Chengkai Li, Xiangpeng Xiao, Yiming Li, Bo Ma, Junwei Mostafavi, Ali Caverlee, James |
| contents | Accurate question answering (QA) in disaster management requires reasoning over uncertain and conflicting information, a setting poorly captured by existing benchmarks built on clean evidence. We introduce DisastQA, a large-scale benchmark of 3,000 rigorously verified questions (2,000 multiple-choice and 1,000 open-ended) spanning eight disaster types. The benchmark is constructed via a human-LLM collaboration pipeline with stratified sampling to ensure balanced coverage. Models are evaluated under varying evidence conditions, from closed-book to noisy evidence integration, enabling separation of internal knowledge from reasoning under imperfect information. For open-ended QA, we propose a human-verified keypoint-based evaluation protocol emphasizing factual completeness over verbosity. Experiments with 20 models reveal substantial divergences from general-purpose leaderboards such as MMLU-Pro. While recent open-weight models approach proprietary systems in clean settings, performance degrades sharply under realistic noise, exposing critical reliability gaps for disaster response. All code, data, and evaluation resources are available at https://github.com/TamuChen18/DisastQA_open. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_03670 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DisastQA: A Comprehensive Benchmark for Evaluating Question Answering in Disaster Management Chen, Zhitong Yin, Kai Dong, Xiangjue Liu, Chengkai Li, Xiangpeng Xiao, Yiming Li, Bo Ma, Junwei Mostafavi, Ali Caverlee, James Computation and Language Accurate question answering (QA) in disaster management requires reasoning over uncertain and conflicting information, a setting poorly captured by existing benchmarks built on clean evidence. We introduce DisastQA, a large-scale benchmark of 3,000 rigorously verified questions (2,000 multiple-choice and 1,000 open-ended) spanning eight disaster types. The benchmark is constructed via a human-LLM collaboration pipeline with stratified sampling to ensure balanced coverage. Models are evaluated under varying evidence conditions, from closed-book to noisy evidence integration, enabling separation of internal knowledge from reasoning under imperfect information. For open-ended QA, we propose a human-verified keypoint-based evaluation protocol emphasizing factual completeness over verbosity. Experiments with 20 models reveal substantial divergences from general-purpose leaderboards such as MMLU-Pro. While recent open-weight models approach proprietary systems in clean settings, performance degrades sharply under realistic noise, exposing critical reliability gaps for disaster response. All code, data, and evaluation resources are available at https://github.com/TamuChen18/DisastQA_open. |
| title | DisastQA: A Comprehensive Benchmark for Evaluating Question Answering in Disaster Management |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.03670 |