$\texttt{COSMIC}$: Mutual Information for Task-Agnostic Summarization Evaluation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909286850887680 |
|---|---|
| author | Darrin, Maxime Formont, Philippe Cheung, Jackie Chi Kit Piantanida, Pablo |
| author_facet | Darrin, Maxime Formont, Philippe Cheung, Jackie Chi Kit Piantanida, Pablo |
| contents | Assessing the quality of summarizers poses significant challenges. In response, we propose a novel task-oriented evaluation approach that assesses summarizers based on their capacity to produce summaries that are useful for downstream tasks, while preserving task outcomes. We theoretically establish a direct relationship between the resulting error probability of these tasks and the mutual information between source texts and generated summaries. We introduce $\texttt{COSMIC}$ as a practical implementation of this metric, demonstrating its strong correlation with human judgment-based metrics and its effectiveness in predicting downstream task performance. Comparative analyses against established metrics like $\texttt{BERTScore}$ and $\texttt{ROUGE}$ highlight the competitive performance of $\texttt{COSMIC}$. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_19457 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | $\texttt{COSMIC}$: Mutual Information for Task-Agnostic Summarization Evaluation Darrin, Maxime Formont, Philippe Cheung, Jackie Chi Kit Piantanida, Pablo Computation and Language Artificial Intelligence Assessing the quality of summarizers poses significant challenges. In response, we propose a novel task-oriented evaluation approach that assesses summarizers based on their capacity to produce summaries that are useful for downstream tasks, while preserving task outcomes. We theoretically establish a direct relationship between the resulting error probability of these tasks and the mutual information between source texts and generated summaries. We introduce $\texttt{COSMIC}$ as a practical implementation of this metric, demonstrating its strong correlation with human judgment-based metrics and its effectiveness in predicting downstream task performance. Comparative analyses against established metrics like $\texttt{BERTScore}$ and $\texttt{ROUGE}$ highlight the competitive performance of $\texttt{COSMIC}$. |
| title | $\texttt{COSMIC}$: Mutual Information for Task-Agnostic Summarization Evaluation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.19457 |