In-context learning enables multimodal large language models to classify cancer pathology images
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917611636260864 |
|---|---|
| author | Ferber, Dyke Wölflein, Georg Wiest, Isabella C. Ligero, Marta Sainath, Srividhya Laleh, Narmin Ghaffari Nahhas, Omar S. M. El Müller-Franzes, Gustav Jäger, Dirk Truhn, Daniel Kather, Jakob Nikolas |
| author_facet | Ferber, Dyke Wölflein, Georg Wiest, Isabella C. Ligero, Marta Sainath, Srividhya Laleh, Narmin Ghaffari Nahhas, Omar S. M. El Müller-Franzes, Gustav Jäger, Dirk Truhn, Daniel Kather, Jakob Nikolas |
| contents | Medical image classification requires labeled, task-specific datasets which are used to train deep learning networks de novo, or to fine-tune foundation models. However, this process is computationally and technically demanding. In language processing, in-context learning provides an alternative, where models learn from within prompts, bypassing the need for parameter updates. Yet, in-context learning remains underexplored in medical image analysis. Here, we systematically evaluate the model Generative Pretrained Transformer 4 with Vision capabilities (GPT-4V) on cancer image processing with in-context learning on three cancer histopathology tasks of high importance: Classification of tissue subtypes in colorectal cancer, colon polyp subtyping and breast tumor detection in lymph node sections. Our results show that in-context learning is sufficient to match or even outperform specialized neural networks trained for particular tasks, while only requiring a minimal number of samples. In summary, this study demonstrates that large vision language models trained on non-domain specific data can be applied out-of-the box to solve medical image-processing tasks in histopathology. This democratizes access of generalist AI models to medical experts without technical background especially for areas where annotated data is scarce. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_07407 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | In-context learning enables multimodal large language models to classify cancer pathology images Ferber, Dyke Wölflein, Georg Wiest, Isabella C. Ligero, Marta Sainath, Srividhya Laleh, Narmin Ghaffari Nahhas, Omar S. M. El Müller-Franzes, Gustav Jäger, Dirk Truhn, Daniel Kather, Jakob Nikolas Computer Vision and Pattern Recognition Medical image classification requires labeled, task-specific datasets which are used to train deep learning networks de novo, or to fine-tune foundation models. However, this process is computationally and technically demanding. In language processing, in-context learning provides an alternative, where models learn from within prompts, bypassing the need for parameter updates. Yet, in-context learning remains underexplored in medical image analysis. Here, we systematically evaluate the model Generative Pretrained Transformer 4 with Vision capabilities (GPT-4V) on cancer image processing with in-context learning on three cancer histopathology tasks of high importance: Classification of tissue subtypes in colorectal cancer, colon polyp subtyping and breast tumor detection in lymph node sections. Our results show that in-context learning is sufficient to match or even outperform specialized neural networks trained for particular tasks, while only requiring a minimal number of samples. In summary, this study demonstrates that large vision language models trained on non-domain specific data can be applied out-of-the box to solve medical image-processing tasks in histopathology. This democratizes access of generalist AI models to medical experts without technical background especially for areas where annotated data is scarce. |
| title | In-context learning enables multimodal large language models to classify cancer pathology images |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.07407 |