LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Description
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912095023398912 |
|---|---|
| author | Jin, Yizhang Li, Jian Zhang, Jiangning Hu, Jianlong Gan, Zhenye Tan, Xin Liu, Yong Wang, Yabiao Wang, Chengjie Ma, Lizhuang |
| author_facet | Jin, Yizhang Li, Jian Zhang, Jiangning Hu, Jianlong Gan, Zhenye Tan, Xin Liu, Yong Wang, Yabiao Wang, Chengjie Ma, Lizhuang |
| contents | Visual Spatial Description (VSD) aims to generate texts that describe the spatial relationships between objects within images. Traditional visual spatial relationship classification (VSRC) methods typically output the spatial relationship between two objects in an image, often neglecting world knowledge and lacking general language capabilities. In this paper, we propose a Large Language-and-Vision Assistant for Visual Spatial Description, named LLaVA-VSD, which is designed for the classification, description, and open-ended description of visual spatial relationships. Specifically, the model first constructs a VSD instruction-following dataset using given figure-caption pairs for the three tasks. It then employs LoRA to fine-tune a Large Language and Vision Assistant for VSD, which has 13 billion parameters and supports high-resolution images. Finally, a large language model (Qwen-2) is used to refine the generated sentences, enhancing their diversity and accuracy. LLaVA-VSD demonstrates excellent multimodal conversational capabilities and can follow open-ended instructions to assist with inquiries about object relationships in images. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_04957 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Description Jin, Yizhang Li, Jian Zhang, Jiangning Hu, Jianlong Gan, Zhenye Tan, Xin Liu, Yong Wang, Yabiao Wang, Chengjie Ma, Lizhuang Computer Vision and Pattern Recognition Artificial Intelligence Visual Spatial Description (VSD) aims to generate texts that describe the spatial relationships between objects within images. Traditional visual spatial relationship classification (VSRC) methods typically output the spatial relationship between two objects in an image, often neglecting world knowledge and lacking general language capabilities. In this paper, we propose a Large Language-and-Vision Assistant for Visual Spatial Description, named LLaVA-VSD, which is designed for the classification, description, and open-ended description of visual spatial relationships. Specifically, the model first constructs a VSD instruction-following dataset using given figure-caption pairs for the three tasks. It then employs LoRA to fine-tune a Large Language and Vision Assistant for VSD, which has 13 billion parameters and supports high-resolution images. Finally, a large language model (Qwen-2) is used to refine the generated sentences, enhancing their diversity and accuracy. LLaVA-VSD demonstrates excellent multimodal conversational capabilities and can follow open-ended instructions to assist with inquiries about object relationships in images. |
| title | LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Description |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2408.04957 |