Segment as You Wish -- Free-Form Language-Based Segmentation for Medical Images
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913915748745216 |
|---|---|
| author | Da, Longchao Wang, Rui Xu, Xiaojian Bhatia, Parminder Kass-Hout, Taha Wei, Hua Xiao, Cao |
| author_facet | Da, Longchao Wang, Rui Xu, Xiaojian Bhatia, Parminder Kass-Hout, Taha Wei, Hua Xiao, Cao |
| contents | Medical imaging is crucial for diagnosing a patient's health condition, and accurate segmentation of these images is essential for isolating regions of interest to ensure precise diagnosis and treatment planning. Existing methods primarily rely on bounding boxes or point-based prompts, while few have explored text-related prompts, despite clinicians often describing their observations and instructions in natural language. To address this gap, we first propose a RAG-based free-form text prompt generator, that leverages the domain corpus to generate diverse and realistic descriptions. Then, we introduce FLanS, a novel medical image segmentation model that handles various free-form text prompts, including professional anatomy-informed queries, anatomy-agnostic position-driven queries, and anatomy-agnostic size-driven queries. Additionally, our model also incorporates a symmetry-aware canonicalization module to ensure consistent, accurate segmentations across varying scan orientations and reduce confusion between the anatomical position of an organ and its appearance in the scan. FLanS is trained on a large-scale dataset of over 100k medical images from 7 public datasets. Comprehensive experiments demonstrate the model's superior language understanding and segmentation precision, along with a deep comprehension of the relationship between them, outperforming SOTA baselines on both in-domain and out-of-domain datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12831 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Segment as You Wish -- Free-Form Language-Based Segmentation for Medical Images Da, Longchao Wang, Rui Xu, Xiaojian Bhatia, Parminder Kass-Hout, Taha Wei, Hua Xiao, Cao Image and Video Processing Artificial Intelligence Computer Vision and Pattern Recognition 68T45, 68U10, 92C55 I.2.7; I.4.9; H.3.3; I.2.6 Medical imaging is crucial for diagnosing a patient's health condition, and accurate segmentation of these images is essential for isolating regions of interest to ensure precise diagnosis and treatment planning. Existing methods primarily rely on bounding boxes or point-based prompts, while few have explored text-related prompts, despite clinicians often describing their observations and instructions in natural language. To address this gap, we first propose a RAG-based free-form text prompt generator, that leverages the domain corpus to generate diverse and realistic descriptions. Then, we introduce FLanS, a novel medical image segmentation model that handles various free-form text prompts, including professional anatomy-informed queries, anatomy-agnostic position-driven queries, and anatomy-agnostic size-driven queries. Additionally, our model also incorporates a symmetry-aware canonicalization module to ensure consistent, accurate segmentations across varying scan orientations and reduce confusion between the anatomical position of an organ and its appearance in the scan. FLanS is trained on a large-scale dataset of over 100k medical images from 7 public datasets. Comprehensive experiments demonstrate the model's superior language understanding and segmentation precision, along with a deep comprehension of the relationship between them, outperforming SOTA baselines on both in-domain and out-of-domain datasets. |
| title | Segment as You Wish -- Free-Form Language-Based Segmentation for Medical Images |
| topic | Image and Video Processing Artificial Intelligence Computer Vision and Pattern Recognition 68T45, 68U10, 92C55 I.2.7; I.4.9; H.3.3; I.2.6 |
| url | https://arxiv.org/abs/2410.12831 |