TARN-VIST: Topic Aware Reinforcement Network for Visual Storytelling
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913269684371456 |
|---|---|
| author | Chen, Weiran Li, Xin Su, Jiaqi Zhu, Guiqian Li, Ying Ji, Yi Liu, Chunping |
| author_facet | Chen, Weiran Li, Xin Su, Jiaqi Zhu, Guiqian Li, Ying Ji, Yi Liu, Chunping |
| contents | As a cross-modal task, visual storytelling aims to generate a story for an ordered image sequence automatically. Different from the image captioning task, visual storytelling requires not only modeling the relationships between objects in the image but also mining the connections between adjacent images. Recent approaches primarily utilize either end-to-end frameworks or multi-stage frameworks to generate relevant stories, but they usually overlook latent topic information. In this paper, in order to generate a more coherent and relevant story, we propose a novel method, Topic Aware Reinforcement Network for VIsual StoryTelling (TARN-VIST). In particular, we pre-extracted the topic information of stories from both visual and linguistic perspectives. Then we apply two topic-consistent reinforcement learning rewards to identify the discrepancy between the generated story and the human-labeled story so as to refine the whole generation process. Extensive experimental results on the VIST dataset and human evaluation demonstrate that our proposed model outperforms most of the competitive models across multiple evaluation metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_11550 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TARN-VIST: Topic Aware Reinforcement Network for Visual Storytelling Chen, Weiran Li, Xin Su, Jiaqi Zhu, Guiqian Li, Ying Ji, Yi Liu, Chunping Computer Vision and Pattern Recognition As a cross-modal task, visual storytelling aims to generate a story for an ordered image sequence automatically. Different from the image captioning task, visual storytelling requires not only modeling the relationships between objects in the image but also mining the connections between adjacent images. Recent approaches primarily utilize either end-to-end frameworks or multi-stage frameworks to generate relevant stories, but they usually overlook latent topic information. In this paper, in order to generate a more coherent and relevant story, we propose a novel method, Topic Aware Reinforcement Network for VIsual StoryTelling (TARN-VIST). In particular, we pre-extracted the topic information of stories from both visual and linguistic perspectives. Then we apply two topic-consistent reinforcement learning rewards to identify the discrepancy between the generated story and the human-labeled story so as to refine the whole generation process. Extensive experimental results on the VIST dataset and human evaluation demonstrate that our proposed model outperforms most of the competitive models across multiple evaluation metrics. |
| title | TARN-VIST: Topic Aware Reinforcement Network for Visual Storytelling |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.11550 |