Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linking
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929636790763520 |
|---|---|
| author | Xu, Zhengfei Zhao, Sijia Hao, Yanchao Liu, Xiaolong Li, Lili Yin, Yuyang Li, Bo Chen, Xi Xin, Xin |
| author_facet | Xu, Zhengfei Zhao, Sijia Hao, Yanchao Liu, Xiaolong Li, Lili Yin, Yuyang Li, Bo Chen, Xi Xin, Xin |
| contents | Visual Entity Linking (VEL) is a crucial task for achieving fine-grained visual understanding, matching objects within images (visual mentions) to entities in a knowledge base. Previous VEL tasks rely on textual inputs, but writing queries for complex scenes can be challenging. Visual inputs like clicks or bounding boxes offer a more convenient alternative. Therefore, we propose a new task, Pixel-Level Visual Entity Linking (PL-VEL), which uses pixel masks from visual inputs to refer to objects, supplementing reference methods for VEL. To facilitate research on this task, we have constructed the MaskOVEN-Wiki dataset through an entirely automatic reverse region-entity annotation framework. This dataset contains over 5 million annotations aligning pixel-level regions with entity-level labels, which will advance visual understanding towards fine-grained. Moreover, as pixel masks correspond to semantic regions in an image, we enhance previous patch-interacted attention with region-interacted attention by a visual semantic tokenization approach. Manual evaluation results indicate that the reverse annotation framework achieved a 94.8% annotation success rate. Experimental results show that models trained on this dataset improved accuracy by 18 points compared to zero-shot models. Additionally, the semantic tokenization method achieved a 5-point accuracy improvement over the trained baseline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13614 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linking Xu, Zhengfei Zhao, Sijia Hao, Yanchao Liu, Xiaolong Li, Lili Yin, Yuyang Li, Bo Chen, Xi Xin, Xin Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Information Retrieval Multimedia Visual Entity Linking (VEL) is a crucial task for achieving fine-grained visual understanding, matching objects within images (visual mentions) to entities in a knowledge base. Previous VEL tasks rely on textual inputs, but writing queries for complex scenes can be challenging. Visual inputs like clicks or bounding boxes offer a more convenient alternative. Therefore, we propose a new task, Pixel-Level Visual Entity Linking (PL-VEL), which uses pixel masks from visual inputs to refer to objects, supplementing reference methods for VEL. To facilitate research on this task, we have constructed the MaskOVEN-Wiki dataset through an entirely automatic reverse region-entity annotation framework. This dataset contains over 5 million annotations aligning pixel-level regions with entity-level labels, which will advance visual understanding towards fine-grained. Moreover, as pixel masks correspond to semantic regions in an image, we enhance previous patch-interacted attention with region-interacted attention by a visual semantic tokenization approach. Manual evaluation results indicate that the reverse annotation framework achieved a 94.8% annotation success rate. Experimental results show that models trained on this dataset improved accuracy by 18 points compared to zero-shot models. Additionally, the semantic tokenization method achieved a 5-point accuracy improvement over the trained baseline. |
| title | Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linking |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Information Retrieval Multimedia |
| url | https://arxiv.org/abs/2412.13614 |