On the Effectiveness of Integration Methods for Multimodal Dialogue Response Retrieval
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913083970027520 |
|---|---|
| author | Jang, Seongbo Lee, Seonghyeon Lee, Dongha Yu, Hwanjo |
| author_facet | Jang, Seongbo Lee, Seonghyeon Lee, Dongha Yu, Hwanjo |
| contents | Multimodal chatbots have become one of the major topics for dialogue systems in both research community and industry. Recently, researchers have shed light on the multimodality of responses as well as dialogue contexts. This work explores how a dialogue system can output responses in various modalities such as text and image. To this end, we first formulate a multimodal dialogue response retrieval task for retrieval-based systems as the combination of three subtasks. We then propose three integration methods based on a two-step approach and an end-to-end approach, and compare the merits and demerits of each method. Experimental results on two datasets demonstrate that the end-to-end approach achieves comparable performance without an intermediate step in the two-step approach. In addition, a parameter sharing strategy not only reduces the number of parameters but also boosts performance by transferring knowledge across the subtasks and the modalities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_11499 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Effectiveness of Integration Methods for Multimodal Dialogue Response Retrieval Jang, Seongbo Lee, Seonghyeon Lee, Dongha Yu, Hwanjo Computation and Language Multimodal chatbots have become one of the major topics for dialogue systems in both research community and industry. Recently, researchers have shed light on the multimodality of responses as well as dialogue contexts. This work explores how a dialogue system can output responses in various modalities such as text and image. To this end, we first formulate a multimodal dialogue response retrieval task for retrieval-based systems as the combination of three subtasks. We then propose three integration methods based on a two-step approach and an end-to-end approach, and compare the merits and demerits of each method. Experimental results on two datasets demonstrate that the end-to-end approach achieves comparable performance without an intermediate step in the two-step approach. In addition, a parameter sharing strategy not only reduces the number of parameters but also boosts performance by transferring knowledge across the subtasks and the modalities. |
| title | On the Effectiveness of Integration Methods for Multimodal Dialogue Response Retrieval |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.11499 |