GroundingGPT:Language Enhanced Multi-modal Grounding Model
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917604740825088 |
|---|---|
| author | Li, Zhaowei Xu, Qi Zhang, Dong Song, Hang Cai, Yiqing Qi, Qi Zhou, Ran Pan, Junting Li, Zefeng Vu, Van Tu Huang, Zhida Wang, Tao |
| author_facet | Li, Zhaowei Xu, Qi Zhang, Dong Song, Hang Cai, Yiqing Qi, Qi Zhou, Ran Pan, Junting Li, Zefeng Vu, Van Tu Huang, Zhida Wang, Tao |
| contents | Multi-modal large language models have demonstrated impressive performance across various tasks in different modalities. However, existing multi-modal models primarily emphasize capturing global information within each modality while neglecting the importance of perceiving local information across modalities. Consequently, these models lack the ability to effectively understand the fine-grained details of input data, limiting their performance in tasks that require a more nuanced understanding. To address this limitation, there is a compelling need to develop models that enable fine-grained understanding across multiple modalities, thereby enhancing their applicability to a wide range of tasks. In this paper, we propose GroundingGPT, a language enhanced multi-modal grounding model. Beyond capturing global information like other multi-modal models, our proposed model excels at tasks demanding a detailed understanding of local information within the input. It demonstrates precise identification and localization of specific regions in images or moments in videos. To achieve this objective, we design a diversified dataset construction pipeline, resulting in a multi-modal, multi-granularity dataset for model training. The code, dataset, and demo of our model can be found at https: //github.com/lzw-lzw/GroundingGPT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_06071 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GroundingGPT:Language Enhanced Multi-modal Grounding Model Li, Zhaowei Xu, Qi Zhang, Dong Song, Hang Cai, Yiqing Qi, Qi Zhou, Ran Pan, Junting Li, Zefeng Vu, Van Tu Huang, Zhida Wang, Tao Computer Vision and Pattern Recognition Computation and Language Multi-modal large language models have demonstrated impressive performance across various tasks in different modalities. However, existing multi-modal models primarily emphasize capturing global information within each modality while neglecting the importance of perceiving local information across modalities. Consequently, these models lack the ability to effectively understand the fine-grained details of input data, limiting their performance in tasks that require a more nuanced understanding. To address this limitation, there is a compelling need to develop models that enable fine-grained understanding across multiple modalities, thereby enhancing their applicability to a wide range of tasks. In this paper, we propose GroundingGPT, a language enhanced multi-modal grounding model. Beyond capturing global information like other multi-modal models, our proposed model excels at tasks demanding a detailed understanding of local information within the input. It demonstrates precise identification and localization of specific regions in images or moments in videos. To achieve this objective, we design a diversified dataset construction pipeline, resulting in a multi-modal, multi-granularity dataset for model training. The code, dataset, and demo of our model can be found at https: //github.com/lzw-lzw/GroundingGPT. |
| title | GroundingGPT:Language Enhanced Multi-modal Grounding Model |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2401.06071 |