Ground-V: Teaching VLMs to Ground Complex Instructions in Pixels
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910955453022208 |
|---|---|
| author | Zong, Yongshuo Zhang, Qin An, Dongsheng Li, Zhihua Xu, Xiang Xu, Linghan Tu, Zhuowen Xing, Yifan Dabeer, Onkar |
| author_facet | Zong, Yongshuo Zhang, Qin An, Dongsheng Li, Zhihua Xu, Xiang Xu, Linghan Tu, Zhuowen Xing, Yifan Dabeer, Onkar |
| contents | This work presents a simple yet effective workflow for automatically scaling instruction-following data to elicit pixel-level grounding capabilities of VLMs under complex instructions. In particular, we address five critical real-world challenges in text-instruction-based grounding: hallucinated references, multi-object scenarios, reasoning, multi-granularity, and part-level references. By leveraging knowledge distillation from a pre-trained teacher model, our approach generates high-quality instruction-response pairs linked to existing pixel-level annotations, minimizing the need for costly human annotation. The resulting dataset, Ground-V, captures rich object localization knowledge and nuanced pixel-level referring expressions. Experiment results show that models trained on Ground-V exhibit substantial improvements across diverse grounding tasks. Specifically, incorporating Ground-V during training directly achieves an average accuracy boost of 4.4% for LISA and a 7.9% for PSALM across six benchmarks on the gIoU metric. It also sets new state-of-the-art results on standard benchmarks such as RefCOCO/+/g. Notably, on gRefCOCO, we achieve an N-Acc of 83.3%, exceeding the previous state-of-the-art by more than 20%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_13788 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Ground-V: Teaching VLMs to Ground Complex Instructions in Pixels Zong, Yongshuo Zhang, Qin An, Dongsheng Li, Zhihua Xu, Xiang Xu, Linghan Tu, Zhuowen Xing, Yifan Dabeer, Onkar Computer Vision and Pattern Recognition This work presents a simple yet effective workflow for automatically scaling instruction-following data to elicit pixel-level grounding capabilities of VLMs under complex instructions. In particular, we address five critical real-world challenges in text-instruction-based grounding: hallucinated references, multi-object scenarios, reasoning, multi-granularity, and part-level references. By leveraging knowledge distillation from a pre-trained teacher model, our approach generates high-quality instruction-response pairs linked to existing pixel-level annotations, minimizing the need for costly human annotation. The resulting dataset, Ground-V, captures rich object localization knowledge and nuanced pixel-level referring expressions. Experiment results show that models trained on Ground-V exhibit substantial improvements across diverse grounding tasks. Specifically, incorporating Ground-V during training directly achieves an average accuracy boost of 4.4% for LISA and a 7.9% for PSALM across six benchmarks on the gIoU metric. It also sets new state-of-the-art results on standard benchmarks such as RefCOCO/+/g. Notably, on gRefCOCO, we achieve an N-Acc of 83.3%, exceeding the previous state-of-the-art by more than 20%. |
| title | Ground-V: Teaching VLMs to Ground Complex Instructions in Pixels |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.13788 |