Generating Faithful and Salient Text from Multimodal Data
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917770204020736 |
|---|---|
| author | Hashem, Tahsina Wang, Weiqing Wijaya, Derry Tanti Ali, Mohammed Eunus Li, Yuan-Fang |
| author_facet | Hashem, Tahsina Wang, Weiqing Wijaya, Derry Tanti Ali, Mohammed Eunus Li, Yuan-Fang |
| contents | While large multimodal models (LMMs) have obtained strong performance on many multimodal tasks, they may still hallucinate while generating text. Their performance on detecting salient features from visual data is also unclear. In this paper, we develop a framework to generate faithful and salient text from mixed-modal data, which includes images and structured data ( represented in knowledge graphs or tables). Specifically, we train a small vision critic model to identify hallucinated and non-salient features from the image modality. The critic model also generates a list of salient image features. This information is used in the post editing step to improve the generation quality. Experiments on two datasets show that our framework improves LMMs' generation quality on both faithfulness and saliency, outperforming recent techniques aimed at reducing hallucination. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_03961 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Generating Faithful and Salient Text from Multimodal Data Hashem, Tahsina Wang, Weiqing Wijaya, Derry Tanti Ali, Mohammed Eunus Li, Yuan-Fang Computer Vision and Pattern Recognition While large multimodal models (LMMs) have obtained strong performance on many multimodal tasks, they may still hallucinate while generating text. Their performance on detecting salient features from visual data is also unclear. In this paper, we develop a framework to generate faithful and salient text from mixed-modal data, which includes images and structured data ( represented in knowledge graphs or tables). Specifically, we train a small vision critic model to identify hallucinated and non-salient features from the image modality. The critic model also generates a list of salient image features. This information is used in the post editing step to improve the generation quality. Experiments on two datasets show that our framework improves LMMs' generation quality on both faithfulness and saliency, outperforming recent techniques aimed at reducing hallucination. |
| title | Generating Faithful and Salient Text from Multimodal Data |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.03961 |