Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.00549 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909630500700160 |
|---|---|
| author | Min, Hyangsuk Lee, Yuho Ban, Minjeong Deng, Jiaqi Kim, Nicole Hee-Yeon Yun, Taewon Su, Hang Cai, Jason Song, Hwanjun |
| author_facet | Min, Hyangsuk Lee, Yuho Ban, Minjeong Deng, Jiaqi Kim, Nicole Hee-Yeon Yun, Taewon Su, Hang Cai, Jason Song, Hwanjun |
| contents | Evaluation frameworks for text summarization have evolved in terms of both domain coverage and metrics. However, existing benchmarks still lack domain-specific assessment criteria, remain predominantly English-centric, and face challenges with human annotation due to the complexity of reasoning. To address these, we introduce MSumBench, which provides a multi-dimensional, multi-domain evaluation of summarization in English and Chinese. It also incorporates specialized assessment criteria for each domain and leverages a multi-agent debate system to enhance annotation quality. By evaluating eight modern summarization models, we discover distinct performance patterns across domains and languages. We further examine large language models as summary evaluators, analyzing the correlation between their evaluation and summarization capabilities, and uncovering systematic bias in their assessment of self-generated summaries. Our benchmark dataset is publicly available at https://github.com/DISL-Lab/MSumBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_00549 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages Min, Hyangsuk Lee, Yuho Ban, Minjeong Deng, Jiaqi Kim, Nicole Hee-Yeon Yun, Taewon Su, Hang Cai, Jason Song, Hwanjun Computation and Language Artificial Intelligence Evaluation frameworks for text summarization have evolved in terms of both domain coverage and metrics. However, existing benchmarks still lack domain-specific assessment criteria, remain predominantly English-centric, and face challenges with human annotation due to the complexity of reasoning. To address these, we introduce MSumBench, which provides a multi-dimensional, multi-domain evaluation of summarization in English and Chinese. It also incorporates specialized assessment criteria for each domain and leverages a multi-agent debate system to enhance annotation quality. By evaluating eight modern summarization models, we discover distinct performance patterns across domains and languages. We further examine large language models as summary evaluators, analyzing the correlation between their evaluation and summarization capabilities, and uncovering systematic bias in their assessment of self-generated summaries. Our benchmark dataset is publicly available at https://github.com/DISL-Lab/MSumBench. |
| title | Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.00549 |