Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911288805818368 |
|---|---|
| author | Zhang, Yue Zuo, Jingxuan Su, Ke Jing, Liqiang |
| author_facet | Zhang, Yue Zuo, Jingxuan Su, Ke Jing, Liqiang |
| contents | Multimodal summarization aims to generate a concise summary based on the input text and image. However, the existing methods potentially suffer from unfactual output. To evaluate the factuality of multimodal summarization models, we propose two fine-grained and explainable evaluation frameworks (FALLACIOUS) for different application scenarios, i.e. reference-based factuality evaluation framework and reference-free factuality evaluation framework. Notably, the reference-free factuality evaluation framework doesn't need ground truth and hence it has a wider application scenario. To evaluate the effectiveness of the proposed frameworks, we compute the correlation between our frameworks and the other metrics. The experimental results show the effectiveness of our proposed method. We will release our code and dataset via github. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_11414 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization Zhang, Yue Zuo, Jingxuan Su, Ke Jing, Liqiang Computation and Language Multimodal summarization aims to generate a concise summary based on the input text and image. However, the existing methods potentially suffer from unfactual output. To evaluate the factuality of multimodal summarization models, we propose two fine-grained and explainable evaluation frameworks (FALLACIOUS) for different application scenarios, i.e. reference-based factuality evaluation framework and reference-free factuality evaluation framework. Notably, the reference-free factuality evaluation framework doesn't need ground truth and hence it has a wider application scenario. To evaluate the effectiveness of the proposed frameworks, we compute the correlation between our frameworks and the other metrics. The experimental results show the effectiveness of our proposed method. We will release our code and dataset via github. |
| title | Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.11414 |