Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916612818337792 |
|---|---|
| author | Koupaee, Mahnaz Vincent, Jake W. Mansour, Saab Shalyminov, Igor He, Han Song, Hwanjun Shu, Raphael He, Jianfeng Nian, Yi Wong, Amy Wing-mei Han, Kyu J. Su, Hang |
| author_facet | Koupaee, Mahnaz Vincent, Jake W. Mansour, Saab Shalyminov, Igor He, Han Song, Hwanjun Shu, Raphael He, Jianfeng Nian, Yi Wong, Amy Wing-mei Han, Kyu J. Su, Hang |
| contents | Faithfulness evaluators based on large language models (LLMs) are often fooled by the fluency of the text and struggle with identifying errors in the summaries. We propose an approach to summary faithfulness evaluation in which multiple LLM-based agents are assigned initial stances (regardless of what their belief might be) and forced to come up with a reason to justify the imposed belief, thus engaging in a multi-round debate to reach an agreement. The uniformly distributed initial assignments result in a greater diversity of stances leading to more meaningful debates and ultimately more errors identified. Furthermore, by analyzing the recent faithfulness evaluation datasets, we observe that naturally, it is not always the case for a summary to be either faithful to the source document or not. We therefore introduce a new dimension, ambiguity, and a detailed taxonomy to identify such special cases. Experiments demonstrate our approach can help identify ambiguities, and have even a stronger performance on non-ambiguous summaries. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_08514 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation Koupaee, Mahnaz Vincent, Jake W. Mansour, Saab Shalyminov, Igor He, Han Song, Hwanjun Shu, Raphael He, Jianfeng Nian, Yi Wong, Amy Wing-mei Han, Kyu J. Su, Hang Computation and Language Faithfulness evaluators based on large language models (LLMs) are often fooled by the fluency of the text and struggle with identifying errors in the summaries. We propose an approach to summary faithfulness evaluation in which multiple LLM-based agents are assigned initial stances (regardless of what their belief might be) and forced to come up with a reason to justify the imposed belief, thus engaging in a multi-round debate to reach an agreement. The uniformly distributed initial assignments result in a greater diversity of stances leading to more meaningful debates and ultimately more errors identified. Furthermore, by analyzing the recent faithfulness evaluation datasets, we observe that naturally, it is not always the case for a summary to be either faithful to the source document or not. We therefore introduce a new dimension, ambiguity, and a detailed taxonomy to identify such special cases. Experiments demonstrate our approach can help identify ambiguities, and have even a stronger performance on non-ambiguous summaries. |
| title | Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.08514 |