WSI-LLaVA: A Multimodal Large Language Model for Whole Slide Image
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915440716939264 |
|---|---|
| author | Liang, Yuci Lyu, Xinheng Chen, Wenting Ding, Meidan Zhang, Jipeng He, Xiangjian Wu, Song Xing, Xiaohan Yang, Sen Wang, Xiyue Shen, Linlin |
| author_facet | Liang, Yuci Lyu, Xinheng Chen, Wenting Ding, Meidan Zhang, Jipeng He, Xiangjian Wu, Song Xing, Xiaohan Yang, Sen Wang, Xiyue Shen, Linlin |
| contents | Recent advancements in computational pathology have produced patch-level Multi-modal Large Language Models (MLLMs), but these models are limited by their inability to analyze whole slide images (WSIs) comprehensively and their tendency to bypass crucial morphological features that pathologists rely on for diagnosis. To address these challenges, we first introduce WSI-Bench, a large-scale morphology-aware benchmark containing 180k VQA pairs from 9,850 WSIs across 30 cancer types, designed to evaluate MLLMs' understanding of morphological characteristics crucial for accurate diagnosis. Building upon this benchmark, we present WSI-LLaVA, a novel framework for gigapixel WSI understanding that employs a three-stage training approach: WSI-text alignment, feature space alignment, and task-specific instruction tuning. To better assess model performance in pathological contexts, we develop two specialized WSI metrics: WSI-Precision and WSI-Relevance. Experimental results demonstrate that WSI-LLaVA outperforms existing models across all capability dimensions, with a significant improvement in morphological analysis, establishing a clear correlation between morphological understanding and diagnostic accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_02141 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WSI-LLaVA: A Multimodal Large Language Model for Whole Slide Image Liang, Yuci Lyu, Xinheng Chen, Wenting Ding, Meidan Zhang, Jipeng He, Xiangjian Wu, Song Xing, Xiaohan Yang, Sen Wang, Xiyue Shen, Linlin Computer Vision and Pattern Recognition Computation and Language Recent advancements in computational pathology have produced patch-level Multi-modal Large Language Models (MLLMs), but these models are limited by their inability to analyze whole slide images (WSIs) comprehensively and their tendency to bypass crucial morphological features that pathologists rely on for diagnosis. To address these challenges, we first introduce WSI-Bench, a large-scale morphology-aware benchmark containing 180k VQA pairs from 9,850 WSIs across 30 cancer types, designed to evaluate MLLMs' understanding of morphological characteristics crucial for accurate diagnosis. Building upon this benchmark, we present WSI-LLaVA, a novel framework for gigapixel WSI understanding that employs a three-stage training approach: WSI-text alignment, feature space alignment, and task-specific instruction tuning. To better assess model performance in pathological contexts, we develop two specialized WSI metrics: WSI-Precision and WSI-Relevance. Experimental results demonstrate that WSI-LLaVA outperforms existing models across all capability dimensions, with a significant improvement in morphological analysis, establishing a clear correlation between morphological understanding and diagnostic accuracy. |
| title | WSI-LLaVA: A Multimodal Large Language Model for Whole Slide Image |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2412.02141 |