Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledge
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916313393266688 |
|---|---|
| author | Lee, Young-Jun Lee, Dokyong Youn, Junyoung Oh, Kyeongjin Ko, Byungsoo Hyeon, Jonghwan Choi, Ho-Jin |
| author_facet | Lee, Young-Jun Lee, Dokyong Youn, Junyoung Oh, Kyeongjin Ko, Byungsoo Hyeon, Jonghwan Choi, Ho-Jin |
| contents | Humans share a wide variety of images related to their personal experiences within conversations via instant messaging tools. However, existing works focus on (1) image-sharing behavior in singular sessions, leading to limited long-term social interaction, and (2) a lack of personalized image-sharing behavior. In this work, we introduce Stark, a large-scale long-term multi-modal conversation dataset that covers a wide range of social personas in a multi-modality format, time intervals, and images. To construct Stark automatically, we propose a novel multi-modal contextualization framework, Mcu, that generates long-term multi-modal dialogue distilled from ChatGPT and our proposed Plan-and-Execute image aligner. Using our Stark, we train a multi-modal conversation model, Ultron 7B, which demonstrates impressive visual imagination ability. Furthermore, we demonstrate the effectiveness of our dataset in human evaluation. We make our source code and dataset publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_03958 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledge Lee, Young-Jun Lee, Dokyong Youn, Junyoung Oh, Kyeongjin Ko, Byungsoo Hyeon, Jonghwan Choi, Ho-Jin Computation and Language Computer Vision and Pattern Recognition Humans share a wide variety of images related to their personal experiences within conversations via instant messaging tools. However, existing works focus on (1) image-sharing behavior in singular sessions, leading to limited long-term social interaction, and (2) a lack of personalized image-sharing behavior. In this work, we introduce Stark, a large-scale long-term multi-modal conversation dataset that covers a wide range of social personas in a multi-modality format, time intervals, and images. To construct Stark automatically, we propose a novel multi-modal contextualization framework, Mcu, that generates long-term multi-modal dialogue distilled from ChatGPT and our proposed Plan-and-Execute image aligner. Using our Stark, we train a multi-modal conversation model, Ultron 7B, which demonstrates impressive visual imagination ability. Furthermore, we demonstrate the effectiveness of our dataset in human evaluation. We make our source code and dataset publicly available. |
| title | Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledge |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.03958 |