BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914910132240384 |
|---|---|
| author | Yoon, Hee Suk Yoon, Eunseop Tee, Joshua Tian Jin Zhang, Kang Heo, Yu-Jung Chang, Du-Seong Yoo, Chang D. |
| author_facet | Yoon, Hee Suk Yoon, Eunseop Tee, Joshua Tian Jin Zhang, Kang Heo, Yu-Jung Chang, Du-Seong Yoo, Chang D. |
| contents | Multimodal Dialogue Response Generation (MDRG) is a recently proposed task where the model needs to generate responses in texts, images, or a blend of both based on the dialogue context. Due to the lack of a large-scale dataset specifically for this task and the benefits of leveraging powerful pre-trained models, previous work relies on the text modality as an intermediary step for both the image input and output of the model rather than adopting an end-to-end approach. However, this approach can overlook crucial information about the image, hindering 1) image-grounded text response and 2) consistency of objects in the image response. In this paper, we propose BI-MDRG that bridges the response generation path such that the image history information is utilized for enhanced relevance of text responses to the image content and the consistency of objects in sequential image responses. Through extensive experiments on the multimodal dialogue benchmark dataset, we show that BI-MDRG can effectively increase the quality of multimodal dialogue. Additionally, recognizing the gap in benchmark datasets for evaluating the image consistency in multimodal dialogue, we have created a curated set of 300 dialogues annotated to track object consistency across conversations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_05926 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generation Yoon, Hee Suk Yoon, Eunseop Tee, Joshua Tian Jin Zhang, Kang Heo, Yu-Jung Chang, Du-Seong Yoo, Chang D. Artificial Intelligence Machine Learning Multimedia Multimodal Dialogue Response Generation (MDRG) is a recently proposed task where the model needs to generate responses in texts, images, or a blend of both based on the dialogue context. Due to the lack of a large-scale dataset specifically for this task and the benefits of leveraging powerful pre-trained models, previous work relies on the text modality as an intermediary step for both the image input and output of the model rather than adopting an end-to-end approach. However, this approach can overlook crucial information about the image, hindering 1) image-grounded text response and 2) consistency of objects in the image response. In this paper, we propose BI-MDRG that bridges the response generation path such that the image history information is utilized for enhanced relevance of text responses to the image content and the consistency of objects in sequential image responses. Through extensive experiments on the multimodal dialogue benchmark dataset, we show that BI-MDRG can effectively increase the quality of multimodal dialogue. Additionally, recognizing the gap in benchmark datasets for evaluating the image consistency in multimodal dialogue, we have created a curated set of 300 dialogues annotated to track object consistency across conversations. |
| title | BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generation |
| topic | Artificial Intelligence Machine Learning Multimedia |
| url | https://arxiv.org/abs/2408.05926 |