BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910852508024832 |
|---|---|
| author | Zhao, Theodore Gu, Yu Yang, Jianwei Usuyama, Naoto Lee, Ho Hin Naumann, Tristan Gao, Jianfeng Crabtree, Angela Abel, Jacob Moung-Wen, Christine Piening, Brian Bifulco, Carlo Wei, Mu Poon, Hoifung Wang, Sheng |
| author_facet | Zhao, Theodore Gu, Yu Yang, Jianwei Usuyama, Naoto Lee, Ho Hin Naumann, Tristan Gao, Jianfeng Crabtree, Angela Abel, Jacob Moung-Wen, Christine Piening, Brian Bifulco, Carlo Wei, Mu Poon, Hoifung Wang, Sheng |
| contents | Biomedical image analysis is fundamental for biomedical discovery in cell biology, pathology, radiology, and many other biomedical domains. Holistic image analysis comprises interdependent subtasks such as segmentation, detection, and recognition of relevant objects. Here, we propose BiomedParse, a biomedical foundation model for imaging parsing that can jointly conduct segmentation, detection, and recognition for 82 object types across 9 imaging modalities. Through joint learning, we can improve accuracy for individual tasks and enable novel applications such as segmenting all relevant objects in an image through a text prompt, rather than requiring users to laboriously specify the bounding box for each object. We leveraged readily available natural-language labels or descriptions accompanying those datasets and use GPT-4 to harmonize the noisy, unstructured text information with established biomedical object ontologies. We created a large dataset comprising over six million triples of image, segmentation mask, and textual description. On image segmentation, we showed that BiomedParse is broadly applicable, outperforming state-of-the-art methods on 102,855 test image-mask-label triples across 9 imaging modalities (everything). On object detection, which aims to locate a specific object of interest, BiomedParse again attained state-of-the-art performance, especially on objects with irregular shapes (everywhere). On object recognition, which aims to identify all objects in a given image along with their semantic types, we showed that BiomedParse can simultaneously segment and label all biomedical objects in an image (all at once). In summary, BiomedParse is an all-in-one tool for biomedical image analysis by jointly solving segmentation, detection, and recognition for all major biomedical image modalities, paving the path for efficient and accurate image-based biomedical discovery. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_12971 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once Zhao, Theodore Gu, Yu Yang, Jianwei Usuyama, Naoto Lee, Ho Hin Naumann, Tristan Gao, Jianfeng Crabtree, Angela Abel, Jacob Moung-Wen, Christine Piening, Brian Bifulco, Carlo Wei, Mu Poon, Hoifung Wang, Sheng Computer Vision and Pattern Recognition Biomedical image analysis is fundamental for biomedical discovery in cell biology, pathology, radiology, and many other biomedical domains. Holistic image analysis comprises interdependent subtasks such as segmentation, detection, and recognition of relevant objects. Here, we propose BiomedParse, a biomedical foundation model for imaging parsing that can jointly conduct segmentation, detection, and recognition for 82 object types across 9 imaging modalities. Through joint learning, we can improve accuracy for individual tasks and enable novel applications such as segmenting all relevant objects in an image through a text prompt, rather than requiring users to laboriously specify the bounding box for each object. We leveraged readily available natural-language labels or descriptions accompanying those datasets and use GPT-4 to harmonize the noisy, unstructured text information with established biomedical object ontologies. We created a large dataset comprising over six million triples of image, segmentation mask, and textual description. On image segmentation, we showed that BiomedParse is broadly applicable, outperforming state-of-the-art methods on 102,855 test image-mask-label triples across 9 imaging modalities (everything). On object detection, which aims to locate a specific object of interest, BiomedParse again attained state-of-the-art performance, especially on objects with irregular shapes (everywhere). On object recognition, which aims to identify all objects in a given image along with their semantic types, we showed that BiomedParse can simultaneously segment and label all biomedical objects in an image (all at once). In summary, BiomedParse is an all-in-one tool for biomedical image analysis by jointly solving segmentation, detection, and recognition for all major biomedical image modalities, paving the path for efficient and accurate image-based biomedical discovery. |
| title | BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.12971 |