Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.07144 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909532607741952 |
|---|---|
| author | Ma, Shengkun Peng, Hao Hou, Lei Li, Juanzi |
| author_facet | Ma, Shengkun Peng, Hao Hou, Lei Li, Juanzi |
| contents | Machine Reading Comprehension (MRC) is an essential task in evaluating natural language understanding. Existing MRC datasets primarily assess specific aspects of reading comprehension (RC), lacking a comprehensive MRC benchmark. To fill this gap, we first introduce a novel taxonomy that categorizes the key capabilities required for RC. Based on this taxonomy, we construct MRCEval, an MRC benchmark that leverages advanced Large Language Models (LLMs) as both sample generators and selection judges. MRCEval is a comprehensive, challenging and accessible benchmark designed to assess the RC capabilities of LLMs thoroughly, covering 13 distinct RC skills with a total of 2.1K high-quality multi-choice questions. We perform an extensive evaluation of 28 widely used open-source and proprietary models, highlighting that MRC continues to present significant challenges even in the era of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_07144 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MRCEval: A Comprehensive, Challenging and Accessible Machine Reading Comprehension Benchmark Ma, Shengkun Peng, Hao Hou, Lei Li, Juanzi Computation and Language Artificial Intelligence Machine Reading Comprehension (MRC) is an essential task in evaluating natural language understanding. Existing MRC datasets primarily assess specific aspects of reading comprehension (RC), lacking a comprehensive MRC benchmark. To fill this gap, we first introduce a novel taxonomy that categorizes the key capabilities required for RC. Based on this taxonomy, we construct MRCEval, an MRC benchmark that leverages advanced Large Language Models (LLMs) as both sample generators and selection judges. MRCEval is a comprehensive, challenging and accessible benchmark designed to assess the RC capabilities of LLMs thoroughly, covering 13 distinct RC skills with a total of 2.1K high-quality multi-choice questions. We perform an extensive evaluation of 28 widely used open-source and proprietary models, highlighting that MRC continues to present significant challenges even in the era of LLMs. |
| title | MRCEval: A Comprehensive, Challenging and Accessible Machine Reading Comprehension Benchmark |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.07144 |