Grounding Multimodal Large Language Models with Quantitative Skin Attributes: A Retrieval Study
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916922727071744 |
|---|---|
| author | Torop, Max Eskandar, Masih Kurtansky, Nicholas Liu, Jinyang Weber, Jochen Camps, Octavia Rotemberg, Veronica Dy, Jennifer Kose, Kivanc |
| author_facet | Torop, Max Eskandar, Masih Kurtansky, Nicholas Liu, Jinyang Weber, Jochen Camps, Octavia Rotemberg, Veronica Dy, Jennifer Kose, Kivanc |
| contents | Artificial Intelligence models have demonstrated significant success in diagnosing skin diseases, including cancer, showing the potential to assist clinicians in their analysis. However, the interpretability of model predictions must be significantly improved before they can be used in practice. To this end, we explore the combination of two promising approaches: Multimodal Large Language Models (MLLMs) and quantitative attribute usage. MLLMs offer a potential avenue for increased interpretability, providing reasoning for diagnosis in natural language through an interactive format. Separately, a number of quantitative attributes that are related to lesion appearance (e.g., lesion area) have recently been found predictive of malignancy with high accuracy. Predictions grounded as a function of such concepts have the potential for improved interpretability. We provide evidence that MLLM embedding spaces can be grounded in such attributes, through fine-tuning to predict their values from images. Concretely, we evaluate this grounding in the embedding space through an attribute-specific content-based image retrieval case study using the SLICE-3D dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_20188 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Grounding Multimodal Large Language Models with Quantitative Skin Attributes: A Retrieval Study Torop, Max Eskandar, Masih Kurtansky, Nicholas Liu, Jinyang Weber, Jochen Camps, Octavia Rotemberg, Veronica Dy, Jennifer Kose, Kivanc Computer Vision and Pattern Recognition Machine Learning Artificial Intelligence models have demonstrated significant success in diagnosing skin diseases, including cancer, showing the potential to assist clinicians in their analysis. However, the interpretability of model predictions must be significantly improved before they can be used in practice. To this end, we explore the combination of two promising approaches: Multimodal Large Language Models (MLLMs) and quantitative attribute usage. MLLMs offer a potential avenue for increased interpretability, providing reasoning for diagnosis in natural language through an interactive format. Separately, a number of quantitative attributes that are related to lesion appearance (e.g., lesion area) have recently been found predictive of malignancy with high accuracy. Predictions grounded as a function of such concepts have the potential for improved interpretability. We provide evidence that MLLM embedding spaces can be grounded in such attributes, through fine-tuning to predict their values from images. Concretely, we evaluate this grounding in the embedding space through an attribute-specific content-based image retrieval case study using the SLICE-3D dataset. |
| title | Grounding Multimodal Large Language Models with Quantitative Skin Attributes: A Retrieval Study |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2508.20188 |