Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.18146 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909232808329216 |
|---|---|
| author | Huang, Xiaoshuang Huang, Haifeng Shen, Lingdong Yang, Yehui Shang, Fangxin Liu, Junwei Liu, Jia |
| author_facet | Huang, Xiaoshuang Huang, Haifeng Shen, Lingdong Yang, Yehui Shang, Fangxin Liu, Junwei Liu, Jia |
| contents | With the rapid development of multimodal large language models (MLLMs), especially their capabilities in visual chat through refer and ground functionalities, their significance is increasingly recognized. However, the biomedical field currently exhibits a substantial gap in this area, primarily due to the absence of a dedicated refer and ground dataset for biomedical images. To address this challenge, we devised the Med-GRIT-270k dataset. It comprises 270k question-and-answer pairs and spans eight distinct medical imaging modalities. Most importantly, it is the first dedicated to the biomedical domain and integrating refer and ground conversations. The key idea is to sample large-scale biomedical image-mask pairs from medical segmentation datasets and generate instruction datasets from text using chatGPT. Additionally, we introduce a Refer-and-Ground Multimodal Large Language Model for Biomedicine (BiRD) by using this dataset and multi-task instruction learning. Extensive experiments have corroborated the efficacy of the Med-GRIT-270k dataset and the multi-modal, fine-grained interactive capabilities of the BiRD model. This holds significant reference value for the exploration and development of intelligent biomedical assistants. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_18146 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Refer-and-Ground Multimodal Large Language Model for Biomedicine Huang, Xiaoshuang Huang, Haifeng Shen, Lingdong Yang, Yehui Shang, Fangxin Liu, Junwei Liu, Jia Computer Vision and Pattern Recognition With the rapid development of multimodal large language models (MLLMs), especially their capabilities in visual chat through refer and ground functionalities, their significance is increasingly recognized. However, the biomedical field currently exhibits a substantial gap in this area, primarily due to the absence of a dedicated refer and ground dataset for biomedical images. To address this challenge, we devised the Med-GRIT-270k dataset. It comprises 270k question-and-answer pairs and spans eight distinct medical imaging modalities. Most importantly, it is the first dedicated to the biomedical domain and integrating refer and ground conversations. The key idea is to sample large-scale biomedical image-mask pairs from medical segmentation datasets and generate instruction datasets from text using chatGPT. Additionally, we introduce a Refer-and-Ground Multimodal Large Language Model for Biomedicine (BiRD) by using this dataset and multi-task instruction learning. Extensive experiments have corroborated the efficacy of the Med-GRIT-270k dataset and the multi-modal, fine-grained interactive capabilities of the BiRD model. This holds significant reference value for the exploration and development of intelligent biomedical assistants. |
| title | A Refer-and-Ground Multimodal Large Language Model for Biomedicine |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.18146 |