InstructDET: Diversifying Referring Object Detection with Generalized Instructions
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909133121257472 |
|---|---|
| author | Dang, Ronghao Feng, Jiangyan Zhang, Haodong Ge, Chongjian Song, Lin Gong, Lijun Liu, Chengju Chen, Qijun Zhu, Feng Zhao, Rui Song, Yibing |
| author_facet | Dang, Ronghao Feng, Jiangyan Zhang, Haodong Ge, Chongjian Song, Lin Gong, Lijun Liu, Chengju Chen, Qijun Zhu, Feng Zhao, Rui Song, Yibing |
| contents | We propose InstructDET, a data-centric method for referring object detection (ROD) that localizes target objects based on user instructions. While deriving from referring expressions (REC), the instructions we leverage are greatly diversified to encompass common user intentions related to object detection. For one image, we produce tremendous instructions that refer to every single object and different combinations of multiple objects. Each instruction and its corresponding object bounding boxes (bbxs) constitute one training data pair. In order to encompass common detection expressions, we involve emerging vision-language model (VLM) and large language model (LLM) to generate instructions guided by text prompts and object bbxs, as the generalizations of foundation models are effective to produce human-like expressions (e.g., describing object property, category, and relationship). We name our constructed dataset as InDET. It contains images, bbxs and generalized instructions that are from foundation models. Our InDET is developed from existing REC datasets and object detection datasets, with the expanding potential that any image with object bbxs can be incorporated through using our InstructDET method. By using our InDET dataset, we show that a conventional ROD model surpasses existing methods on standard REC datasets and our InDET test set. Our data-centric method InstructDET, with automatic data expansion by leveraging foundation models, directs a promising field that ROD can be greatly diversified to execute common object detection instructions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_05136 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | InstructDET: Diversifying Referring Object Detection with Generalized Instructions Dang, Ronghao Feng, Jiangyan Zhang, Haodong Ge, Chongjian Song, Lin Gong, Lijun Liu, Chengju Chen, Qijun Zhu, Feng Zhao, Rui Song, Yibing Artificial Intelligence Computer Vision and Pattern Recognition We propose InstructDET, a data-centric method for referring object detection (ROD) that localizes target objects based on user instructions. While deriving from referring expressions (REC), the instructions we leverage are greatly diversified to encompass common user intentions related to object detection. For one image, we produce tremendous instructions that refer to every single object and different combinations of multiple objects. Each instruction and its corresponding object bounding boxes (bbxs) constitute one training data pair. In order to encompass common detection expressions, we involve emerging vision-language model (VLM) and large language model (LLM) to generate instructions guided by text prompts and object bbxs, as the generalizations of foundation models are effective to produce human-like expressions (e.g., describing object property, category, and relationship). We name our constructed dataset as InDET. It contains images, bbxs and generalized instructions that are from foundation models. Our InDET is developed from existing REC datasets and object detection datasets, with the expanding potential that any image with object bbxs can be incorporated through using our InstructDET method. By using our InDET dataset, we show that a conventional ROD model surpasses existing methods on standard REC datasets and our InDET test set. Our data-centric method InstructDET, with automatic data expansion by leveraging foundation models, directs a promising field that ROD can be greatly diversified to execute common object detection instructions. |
| title | InstructDET: Diversifying Referring Object Detection with Generalized Instructions |
| topic | Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2310.05136 |