QAPyramid: Fine-grained Evaluation of Content Selection for Text Summarization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915535372943360 |
|---|---|
| author | Zhang, Shiyue Wan, David Cattan, Arie Klein, Ayal Dagan, Ido Bansal, Mohit |
| author_facet | Zhang, Shiyue Wan, David Cattan, Arie Klein, Ayal Dagan, Ido Bansal, Mohit |
| contents | How to properly conduct human evaluations for text summarization is a longstanding challenge. The Pyramid human evaluation protocol, which assesses content selection by breaking the reference summary into subunits and verifying their presence in the system summary, has been widely adopted. However, it suffers from a lack of systematicity in the definition and granularity of the sub-units. We address these problems by proposing QAPyramid, which decomposes each reference summary into finer-grained question-answer (QA) pairs according to the QA-SRL framework. We collect QA-SRL annotations for reference summaries from CNN/DM and evaluate 10 summarization systems, resulting in 8.9K QA-level annotations. We show that, compared to Pyramid, QAPyramid provides more systematic and fine-grained content selection evaluation while maintaining high inter-annotator agreement without needing expert annotations. Furthermore, we propose metrics that automate the evaluation pipeline and achieve higher correlations with QAPyramid than other widely adopted metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_07096 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | QAPyramid: Fine-grained Evaluation of Content Selection for Text Summarization Zhang, Shiyue Wan, David Cattan, Arie Klein, Ayal Dagan, Ido Bansal, Mohit Computation and Language Artificial Intelligence How to properly conduct human evaluations for text summarization is a longstanding challenge. The Pyramid human evaluation protocol, which assesses content selection by breaking the reference summary into subunits and verifying their presence in the system summary, has been widely adopted. However, it suffers from a lack of systematicity in the definition and granularity of the sub-units. We address these problems by proposing QAPyramid, which decomposes each reference summary into finer-grained question-answer (QA) pairs according to the QA-SRL framework. We collect QA-SRL annotations for reference summaries from CNN/DM and evaluate 10 summarization systems, resulting in 8.9K QA-level annotations. We show that, compared to Pyramid, QAPyramid provides more systematic and fine-grained content selection evaluation while maintaining high inter-annotator agreement without needing expert annotations. Furthermore, we propose metrics that automate the evaluation pipeline and achieve higher correlations with QAPyramid than other widely adopted metrics. |
| title | QAPyramid: Fine-grained Evaluation of Content Selection for Text Summarization |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.07096 |