VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915556260577280 |
|---|---|
| author | Hoopes, Andrew Dey, Neel Butoi, Victor Ion Guttag, John V. Dalca, Adrian V. |
| author_facet | Hoopes, Andrew Dey, Neel Butoi, Victor Ion Guttag, John V. Dalca, Adrian V. |
| contents | We present VoxelPrompt, an end-to-end image analysis agent that tackles free-form radiological tasks. Given any number of volumetric medical images and a natural language prompt, VoxelPrompt integrates a language model that generates executable code to invoke a jointly-trained, adaptable vision network. This code further carries out analytical steps to address practical quantitative aims, such as measuring the growth of a tumor across visits. The pipelines generated by VoxelPrompt automate analyses that currently require practitioners to painstakingly combine multiple specialized vision and statistical tools. We evaluate VoxelPrompt using diverse neuroimaging tasks and show that it can delineate hundreds of anatomical and pathological features, measure complex morphological properties, and perform open-language analysis of lesion characteristics. VoxelPrompt performs these objectives with an accuracy similar to that of specialist single-task models for image analysis, while facilitating a broad range of compositional biomedical workflows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_08397 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysis Hoopes, Andrew Dey, Neel Butoi, Victor Ion Guttag, John V. Dalca, Adrian V. Image and Video Processing Artificial Intelligence Computer Vision and Pattern Recognition We present VoxelPrompt, an end-to-end image analysis agent that tackles free-form radiological tasks. Given any number of volumetric medical images and a natural language prompt, VoxelPrompt integrates a language model that generates executable code to invoke a jointly-trained, adaptable vision network. This code further carries out analytical steps to address practical quantitative aims, such as measuring the growth of a tumor across visits. The pipelines generated by VoxelPrompt automate analyses that currently require practitioners to painstakingly combine multiple specialized vision and statistical tools. We evaluate VoxelPrompt using diverse neuroimaging tasks and show that it can delineate hundreds of anatomical and pathological features, measure complex morphological properties, and perform open-language analysis of lesion characteristics. VoxelPrompt performs these objectives with an accuracy similar to that of specialist single-task models for image analysis, while facilitating a broad range of compositional biomedical workflows. |
| title | VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysis |
| topic | Image and Video Processing Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.08397 |