Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2311.07884 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909155262988288 |
|---|---|
| author | Zhang, Yusen Zhang, Nan Liu, Yixin Fabbri, Alexander Liu, Junru Kamoi, Ryo Lu, Xiaoxin Xiong, Caiming Zhao, Jieyu Radev, Dragomir McKeown, Kathleen Zhang, Rui |
| author_facet | Zhang, Yusen Zhang, Nan Liu, Yixin Fabbri, Alexander Liu, Junru Kamoi, Ryo Lu, Xiaoxin Xiong, Caiming Zhao, Jieyu Radev, Dragomir McKeown, Kathleen Zhang, Rui |
| contents | People from different social and demographic groups express diverse perspectives and conflicting opinions on a broad set of topics such as product reviews, healthcare, law, and politics. A fair summary should provide a comprehensive coverage of diverse perspectives without underrepresenting certain groups. However, current work in summarization metrics and Large Language Models (LLMs) evaluation has not explored fair abstractive summarization. In this paper, we systematically investigate fair abstractive summarization for user-generated data. We first formally define fairness in abstractive summarization as not underrepresenting perspectives of any groups of people, and we propose four reference-free automatic metrics by measuring the differences between target and source perspectives. We evaluate nine LLMs, including three GPT models, four LLaMA models, PaLM 2, and Claude, on six datasets collected from social media, online reviews, and recorded transcripts. Experiments show that both the model-generated and the human-written reference summaries suffer from low fairness. We conduct a comprehensive analysis of the common factors influencing fairness and propose three simple but effective methods to alleviate unfair summarization. Our dataset and code are available at https://github.com/psunlpgroup/FairSumm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_07884 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Fair Abstractive Summarization of Diverse Perspectives Zhang, Yusen Zhang, Nan Liu, Yixin Fabbri, Alexander Liu, Junru Kamoi, Ryo Lu, Xiaoxin Xiong, Caiming Zhao, Jieyu Radev, Dragomir McKeown, Kathleen Zhang, Rui Computation and Language People from different social and demographic groups express diverse perspectives and conflicting opinions on a broad set of topics such as product reviews, healthcare, law, and politics. A fair summary should provide a comprehensive coverage of diverse perspectives without underrepresenting certain groups. However, current work in summarization metrics and Large Language Models (LLMs) evaluation has not explored fair abstractive summarization. In this paper, we systematically investigate fair abstractive summarization for user-generated data. We first formally define fairness in abstractive summarization as not underrepresenting perspectives of any groups of people, and we propose four reference-free automatic metrics by measuring the differences between target and source perspectives. We evaluate nine LLMs, including three GPT models, four LLaMA models, PaLM 2, and Claude, on six datasets collected from social media, online reviews, and recorded transcripts. Experiments show that both the model-generated and the human-written reference summaries suffer from low fairness. We conduct a comprehensive analysis of the common factors influencing fairness and propose three simple but effective methods to alleviate unfair summarization. Our dataset and code are available at https://github.com/psunlpgroup/FairSumm. |
| title | Fair Abstractive Summarization of Diverse Perspectives |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2311.07884 |