Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929254445350912 |
|---|---|
| author | Song, Jongyoon Park, Nohil Hwang, Bongkyu Yun, Jaewoong Joe, Seongho Gwon, Youngjune L. Yoon, Sungroh |
| author_facet | Song, Jongyoon Park, Nohil Hwang, Bongkyu Yun, Jaewoong Joe, Seongho Gwon, Youngjune L. Yoon, Sungroh |
| contents | Abstractive summarization models often generate factually inconsistent content particularly when the parametric knowledge of the model conflicts with the knowledge in the input document. In this paper, we analyze the robustness of fine-tuning based summarization models to the knowledge conflict, which we call factual adaptiveness. We utilize pre-trained language models to construct evaluation sets and find that factual adaptiveness is not strongly correlated with factual consistency on original datasets. Furthermore, we introduce a controllable counterfactual data augmentation method where the degree of knowledge conflict within the augmented data can be adjustable. Our experimental results on two pre-trained language models (PEGASUS and BART) and two fine-tuning datasets (XSum and CNN/DailyMail) demonstrate that our method enhances factual adaptiveness while achieving factual consistency on original datasets on par with the contrastive learning baseline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_15162 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Models Song, Jongyoon Park, Nohil Hwang, Bongkyu Yun, Jaewoong Joe, Seongho Gwon, Youngjune L. Yoon, Sungroh Computation and Language Artificial Intelligence Machine Learning Abstractive summarization models often generate factually inconsistent content particularly when the parametric knowledge of the model conflicts with the knowledge in the input document. In this paper, we analyze the robustness of fine-tuning based summarization models to the knowledge conflict, which we call factual adaptiveness. We utilize pre-trained language models to construct evaluation sets and find that factual adaptiveness is not strongly correlated with factual consistency on original datasets. Furthermore, we introduce a controllable counterfactual data augmentation method where the degree of knowledge conflict within the augmented data can be adjustable. Our experimental results on two pre-trained language models (PEGASUS and BART) and two fine-tuning datasets (XSum and CNN/DailyMail) demonstrate that our method enhances factual adaptiveness while achieving factual consistency on original datasets on par with the contrastive learning baseline. |
| title | Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2402.15162 |