Generalizable Entity Grounding via Assistance of Large Language Model
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909094919536640 |
|---|---|
| author | Qi, Lu Chen, Yi-Wen Yang, Lehan Shen, Tiancheng Li, Xiangtai Guo, Weidong Xu, Yu Yang, Ming-Hsuan |
| author_facet | Qi, Lu Chen, Yi-Wen Yang, Lehan Shen, Tiancheng Li, Xiangtai Guo, Weidong Xu, Yu Yang, Ming-Hsuan |
| contents | In this work, we propose a novel approach to densely ground visual entities from a long caption. We leverage a large multimodal model (LMM) to extract semantic nouns, a class-agnostic segmentation model to generate entity-level segmentation, and the proposed multi-modal feature fusion module to associate each semantic noun with its corresponding segmentation mask. Additionally, we introduce a strategy of encoding entity segmentation masks into a colormap, enabling the preservation of fine-grained predictions from features of high-resolution masks. This approach allows us to extract visual features from low-resolution images using the CLIP vision encoder in the LMM, which is more computationally efficient than existing approaches that use an additional encoder for high-resolution images. Our comprehensive experiments demonstrate the superiority of our method, outperforming state-of-the-art techniques on three tasks, including panoptic narrative grounding, referring expression segmentation, and panoptic segmentation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_02555 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Generalizable Entity Grounding via Assistance of Large Language Model Qi, Lu Chen, Yi-Wen Yang, Lehan Shen, Tiancheng Li, Xiangtai Guo, Weidong Xu, Yu Yang, Ming-Hsuan Computer Vision and Pattern Recognition Computation and Language In this work, we propose a novel approach to densely ground visual entities from a long caption. We leverage a large multimodal model (LMM) to extract semantic nouns, a class-agnostic segmentation model to generate entity-level segmentation, and the proposed multi-modal feature fusion module to associate each semantic noun with its corresponding segmentation mask. Additionally, we introduce a strategy of encoding entity segmentation masks into a colormap, enabling the preservation of fine-grained predictions from features of high-resolution masks. This approach allows us to extract visual features from low-resolution images using the CLIP vision encoder in the LMM, which is more computationally efficient than existing approaches that use an additional encoder for high-resolution images. Our comprehensive experiments demonstrate the superiority of our method, outperforming state-of-the-art techniques on three tasks, including panoptic narrative grounding, referring expression segmentation, and panoptic segmentation. |
| title | Generalizable Entity Grounding via Assistance of Large Language Model |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2402.02555 |