CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909317834211328 |
|---|---|
| author | Gong, Ziwei Ai, Lin Deshpande, Harshsaiprasad Johnson, Alexander Phung, Emmy Wu, Zehui Emami, Ahmad Hirschberg, Julia |
| author_facet | Gong, Ziwei Ai, Lin Deshpande, Harshsaiprasad Johnson, Alexander Phung, Emmy Wu, Zehui Emami, Ahmad Hirschberg, Julia |
| contents | Large Language Models (LLMs) have spurred interest in automatic evaluation methods for summarization, offering a faster, more cost-effective alternative to human evaluation. However, existing methods often fall short when applied to complex tasks like long-context summarizations and dialogue-based meeting summarizations. In this paper, we introduce CREAM (Comparison-Based Reference-Free Elo-Ranked Automatic Evaluation for Meeting Summarization), a novel framework that addresses the unique challenges of evaluating meeting summaries. CREAM leverages a combination of chain-of-thought reasoning and key facts alignment to assess conciseness and completeness of model-generated summaries without requiring reference. By employing an ELO ranking system, our approach provides a robust mechanism for comparing the quality of different models or prompt configurations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_10883 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarization Gong, Ziwei Ai, Lin Deshpande, Harshsaiprasad Johnson, Alexander Phung, Emmy Wu, Zehui Emami, Ahmad Hirschberg, Julia Computation and Language Large Language Models (LLMs) have spurred interest in automatic evaluation methods for summarization, offering a faster, more cost-effective alternative to human evaluation. However, existing methods often fall short when applied to complex tasks like long-context summarizations and dialogue-based meeting summarizations. In this paper, we introduce CREAM (Comparison-Based Reference-Free Elo-Ranked Automatic Evaluation for Meeting Summarization), a novel framework that addresses the unique challenges of evaluating meeting summaries. CREAM leverages a combination of chain-of-thought reasoning and key facts alignment to assess conciseness and completeness of model-generated summaries without requiring reference. By employing an ELO ranking system, our approach provides a robust mechanism for comparing the quality of different models or prompt configurations. |
| title | CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.10883 |