CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909317834211328 |
|---|---|
| author | Gong, Ziwei Ai, Lin Deshpande, Harshsaiprasad Johnson, Alexander Phung, Emmy Wu, Zehui Emami, Ahmad Hirschberg, Julia |
| author_facet | Gong, Ziwei Ai, Lin Deshpande, Harshsaiprasad Johnson, Alexander Phung, Emmy Wu, Zehui Emami, Ahmad Hirschberg, Julia |
| contents | Large Language Models (LLMs) have spurred interest in automatic evaluation methods for summarization, offering a faster, more cost-effective alternative to human evaluation. However, existing methods often fall short when applied to complex tasks like long-context summarizations and dialogue-based meeting summarizations. In this paper, we introduce CREAM (Comparison-Based Reference-Free Elo-Ranked Automatic Evaluation for Meeting Summarization), a novel framework that addresses the unique challenges of evaluating meeting summaries. CREAM leverages a combination of chain-of-thought reasoning and key facts alignment to assess conciseness and completeness of model-generated summaries without requiring reference. By employing an ELO ranking system, our approach provides a robust mechanism for comparing the quality of different models or prompt configurations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_10883 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarization Gong, Ziwei Ai, Lin Deshpande, Harshsaiprasad Johnson, Alexander Phung, Emmy Wu, Zehui Emami, Ahmad Hirschberg, Julia Computation and Language Large Language Models (LLMs) have spurred interest in automatic evaluation methods for summarization, offering a faster, more cost-effective alternative to human evaluation. However, existing methods often fall short when applied to complex tasks like long-context summarizations and dialogue-based meeting summarizations. In this paper, we introduce CREAM (Comparison-Based Reference-Free Elo-Ranked Automatic Evaluation for Meeting Summarization), a novel framework that addresses the unique challenges of evaluating meeting summaries. CREAM leverages a combination of chain-of-thought reasoning and key facts alignment to assess conciseness and completeness of model-generated summaries without requiring reference. By employing an ELO ranking system, our approach provides a robust mechanism for comparing the quality of different models or prompt configurations. |
| title | CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.10883 |