Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910001329602560 |
|---|---|
| author | Sun, Yanpeng Hao, Jing Zhu, Ke Liu, Jiang-Jiang Zhao, Yuxiang Li, Xiaofan Zhao, Na Li, Zechao Wang, Jingdong |
| author_facet | Sun, Yanpeng Hao, Jing Zhu, Ke Liu, Jiang-Jiang Zhao, Yuxiang Li, Xiaofan Zhao, Na Li, Zechao Wang, Jingdong |
| contents | Training Large Multimodality Models (LMMs) relies on descriptive image caption that connects image and language. Existing methods for generating such captions often rely on distilling the captions from pretrained LMMs, constructing them from publicly available internet images, or even generating them through human annotation. However, these strategies can fall short in terms of precision and granularity, particularly when dealing with complex visual reasoning tasks. In this paper, we propose to leverage off-the-shelf visual specialists, which were trained from annotated images initially not for image captioning, for enhancing the image caption. Our approach, named EDC, explores object low-level and fine-grained attributes (e.g., depth, emotion and fine-grained categories) and object relations (e.g., relative location and human-object-interaction (HOI)), and combine the attributes into the descriptive caption. By systematically integrating these rich attributes into the generated captions, EDC significantly improves the descriptive quality of the captions, providing a deeper and more nuanced understanding of the visual content. Experiments demonstrate that such visual specialists are able to improve the performance for visual understanding tasks as well as reasoning that benefits from more accurate visual understanding. The complete source code of EDC pipeline and datasets will be available at https://github.com/syp2ysy/DCE. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_14233 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception Sun, Yanpeng Hao, Jing Zhu, Ke Liu, Jiang-Jiang Zhao, Yuxiang Li, Xiaofan Zhao, Na Li, Zechao Wang, Jingdong Computer Vision and Pattern Recognition Training Large Multimodality Models (LMMs) relies on descriptive image caption that connects image and language. Existing methods for generating such captions often rely on distilling the captions from pretrained LMMs, constructing them from publicly available internet images, or even generating them through human annotation. However, these strategies can fall short in terms of precision and granularity, particularly when dealing with complex visual reasoning tasks. In this paper, we propose to leverage off-the-shelf visual specialists, which were trained from annotated images initially not for image captioning, for enhancing the image caption. Our approach, named EDC, explores object low-level and fine-grained attributes (e.g., depth, emotion and fine-grained categories) and object relations (e.g., relative location and human-object-interaction (HOI)), and combine the attributes into the descriptive caption. By systematically integrating these rich attributes into the generated captions, EDC significantly improves the descriptive quality of the captions, providing a deeper and more nuanced understanding of the visual content. Experiments demonstrate that such visual specialists are able to improve the performance for visual understanding tasks as well as reasoning that benefits from more accurate visual understanding. The complete source code of EDC pipeline and datasets will be available at https://github.com/syp2ysy/DCE. |
| title | Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.14233 |