Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.15759 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910647311138816 |
|---|---|
| author | Jiang, Zekun Cheng, Dongjie Qin, Ziyuan Gao, Jun Lao, Qicheng Ismoilovich, Abdullaev Bakhrom Gayrat, Urazboev Elyorbek, Yuldashov Habibullo, Bekchanov Tang, Defu Wei, LinJing Li, Kang Zhang, Le |
| author_facet | Jiang, Zekun Cheng, Dongjie Qin, Ziyuan Gao, Jun Lao, Qicheng Ismoilovich, Abdullaev Bakhrom Gayrat, Urazboev Elyorbek, Yuldashov Habibullo, Bekchanov Tang, Defu Wei, LinJing Li, Kang Zhang, Le |
| contents | This study presents a novel multimodal medical image zero-shot segmentation algorithm named the text-visual-prompt segment anything model (TV-SAM) without any manual annotations. The TV-SAM incorporates and integrates the large language model GPT-4, the vision language model GLIP, and the SAM to autonomously generate descriptive text prompts and visual bounding box prompts from medical images, thereby enhancing the SAM's capability for zero-shot segmentation. Comprehensive evaluations are implemented on seven public datasets encompassing eight imaging modalities to demonstrate that TV-SAM can effectively segment unseen targets across various modalities without additional training. TV-SAM significantly outperforms SAM AUTO and GSAM, closely matching the performance of SAM BBOX with gold standard bounding box prompts and surpasses the state-of-the-art methods on specific datasets such as ISIC and WBC. The study indicates that TV-SAM serves as an effective multimodal medical image zero-shot segmentation algorithm, highlighting the significant contribution of GPT-4 to zero-shot segmentation. By integrating foundational models such as GPT-4, GLIP, and SAM, the ability to address complex problems in specialized domains can be enhanced. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_15759 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TV-SAM: Increasing Zero-Shot Segmentation Performance on Multimodal Medical Images Using GPT-4 Generated Descriptive Prompts Without Human Annotation Jiang, Zekun Cheng, Dongjie Qin, Ziyuan Gao, Jun Lao, Qicheng Ismoilovich, Abdullaev Bakhrom Gayrat, Urazboev Elyorbek, Yuldashov Habibullo, Bekchanov Tang, Defu Wei, LinJing Li, Kang Zhang, Le Computer Vision and Pattern Recognition Artificial Intelligence This study presents a novel multimodal medical image zero-shot segmentation algorithm named the text-visual-prompt segment anything model (TV-SAM) without any manual annotations. The TV-SAM incorporates and integrates the large language model GPT-4, the vision language model GLIP, and the SAM to autonomously generate descriptive text prompts and visual bounding box prompts from medical images, thereby enhancing the SAM's capability for zero-shot segmentation. Comprehensive evaluations are implemented on seven public datasets encompassing eight imaging modalities to demonstrate that TV-SAM can effectively segment unseen targets across various modalities without additional training. TV-SAM significantly outperforms SAM AUTO and GSAM, closely matching the performance of SAM BBOX with gold standard bounding box prompts and surpasses the state-of-the-art methods on specific datasets such as ISIC and WBC. The study indicates that TV-SAM serves as an effective multimodal medical image zero-shot segmentation algorithm, highlighting the significant contribution of GPT-4 to zero-shot segmentation. By integrating foundational models such as GPT-4, GLIP, and SAM, the ability to address complex problems in specialized domains can be enhanced. |
| title | TV-SAM: Increasing Zero-Shot Segmentation Performance on Multimodal Medical Images Using GPT-4 Generated Descriptive Prompts Without Human Annotation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2402.15759 |