Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2309.15516 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909098503569408 |
|---|---|
| author | Sun, Xiaowen Feng, Jiazhan Wang, Yuxuan Lai, Yuxuan Shen, Xingyu Zhao, Dongyan |
| author_facet | Sun, Xiaowen Feng, Jiazhan Wang, Yuxuan Lai, Yuxuan Shen, Xingyu Zhao, Dongyan |
| contents | A picture is worth a thousand words, thus, it is crucial for conversational agents to understand, perceive, and effectively respond with pictures. However, we find that directly employing conventional image generation techniques is inadequate for conversational agents to produce image responses effectively. In this paper, we focus on the innovative dialog-to-image generation task, where the model synthesizes a high-resolution image aligned with the given dialog context as a response. To tackle this problem, we design a tailored fine-tuning approach on the top of state-of-the-art text-to-image generation models to fully exploit the structural and semantic features in dialog context during image generation. Concretely, we linearize the dialog context with specific indicators to maintain the dialog structure, and employ in-domain data to alleviate the style mismatch between dialog-to-image and conventional image generation tasks. Empirical results on PhotoChat and MMDialog Corpus show that our approach brings consistent and remarkable improvement with 3 state-of-the-art pre-trained text-to-image generation backbones. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_15516 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Teaching Text-to-Image Models to Communicate in Dialog Sun, Xiaowen Feng, Jiazhan Wang, Yuxuan Lai, Yuxuan Shen, Xingyu Zhao, Dongyan Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition A picture is worth a thousand words, thus, it is crucial for conversational agents to understand, perceive, and effectively respond with pictures. However, we find that directly employing conventional image generation techniques is inadequate for conversational agents to produce image responses effectively. In this paper, we focus on the innovative dialog-to-image generation task, where the model synthesizes a high-resolution image aligned with the given dialog context as a response. To tackle this problem, we design a tailored fine-tuning approach on the top of state-of-the-art text-to-image generation models to fully exploit the structural and semantic features in dialog context during image generation. Concretely, we linearize the dialog context with specific indicators to maintain the dialog structure, and employ in-domain data to alleviate the style mismatch between dialog-to-image and conventional image generation tasks. Empirical results on PhotoChat and MMDialog Corpus show that our approach brings consistent and remarkable improvement with 3 state-of-the-art pre-trained text-to-image generation backbones. |
| title | Teaching Text-to-Image Models to Communicate in Dialog |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2309.15516 |