A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913997378289664 |
|---|---|
| author | Xu, Zhe Liu, Ziyi Hou, Junlin Ma, Jiabo Jin, Cheng Wang, Yihui Chen, Zhixuan Zhang, Zhengyu Huang, Fuxiang Guo, Zhengrui Zhou, Fengtao Xu, Yingxue Wang, Xi Chan, Ronald Cheong Kin Liang, Li Chen, Hao |
| author_facet | Xu, Zhe Liu, Ziyi Hou, Junlin Ma, Jiabo Jin, Cheng Wang, Yihui Chen, Zhixuan Zhang, Zhengyu Huang, Fuxiang Guo, Zhengrui Zhou, Fengtao Xu, Yingxue Wang, Xi Chan, Ronald Cheong Kin Liang, Li Chen, Hao |
| contents | Multimodal large language models (MLLMs) have emerged as powerful tools for computational pathology, offering unprecedented opportunities to integrate pathological images with language context for comprehensive diagnostic analysis. These models hold particular promise for automating complex tasks that traditionally require expert interpretation of pathologists. However, current MLLM approaches in pathology demonstrate significantly constrained reasoning capabilities, primarily due to their reliance on expensive chain-of-thought annotations. Additionally, existing methods remain limited to simplex application of visual question answering (VQA) at the region-of-interest (ROI) level, failing to address the full spectrum of diagnostic needs such as ROI classification, detection, segmentation, whole-slide-image (WSI) classification and VQA in clinical practice. In this study, we present SmartPath-R1, a versatile MLLM capable of simultaneously addressing both ROI-level and WSI-level tasks while demonstrating robust pathological reasoning capability. Our framework combines scale-dependent supervised fine-tuning and task-aware reinforcement fine-tuning, which circumvents the requirement for chain-of-thought supervision by leveraging the intrinsic knowledge within MLLM. Furthermore, SmartPath-R1 integrates multiscale and multitask analysis through a mixture-of-experts mechanism, enabling dynamic processing for diverse tasks. We curate a large-scale dataset comprising 2.3M ROI samples and 188K WSI samples for training and evaluation. Extensive experiments across 72 tasks validate the effectiveness and superiority of the proposed approach. This work represents a significant step toward developing versatile, reasoning-enhanced AI systems for precision pathology. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_17303 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model Xu, Zhe Liu, Ziyi Hou, Junlin Ma, Jiabo Jin, Cheng Wang, Yihui Chen, Zhixuan Zhang, Zhengyu Huang, Fuxiang Guo, Zhengrui Zhou, Fengtao Xu, Yingxue Wang, Xi Chan, Ronald Cheong Kin Liang, Li Chen, Hao Image and Video Processing Artificial Intelligence Computer Vision and Pattern Recognition Multimodal large language models (MLLMs) have emerged as powerful tools for computational pathology, offering unprecedented opportunities to integrate pathological images with language context for comprehensive diagnostic analysis. These models hold particular promise for automating complex tasks that traditionally require expert interpretation of pathologists. However, current MLLM approaches in pathology demonstrate significantly constrained reasoning capabilities, primarily due to their reliance on expensive chain-of-thought annotations. Additionally, existing methods remain limited to simplex application of visual question answering (VQA) at the region-of-interest (ROI) level, failing to address the full spectrum of diagnostic needs such as ROI classification, detection, segmentation, whole-slide-image (WSI) classification and VQA in clinical practice. In this study, we present SmartPath-R1, a versatile MLLM capable of simultaneously addressing both ROI-level and WSI-level tasks while demonstrating robust pathological reasoning capability. Our framework combines scale-dependent supervised fine-tuning and task-aware reinforcement fine-tuning, which circumvents the requirement for chain-of-thought supervision by leveraging the intrinsic knowledge within MLLM. Furthermore, SmartPath-R1 integrates multiscale and multitask analysis through a mixture-of-experts mechanism, enabling dynamic processing for diverse tasks. We curate a large-scale dataset comprising 2.3M ROI samples and 188K WSI samples for training and evaluation. Extensive experiments across 72 tasks validate the effectiveness and superiority of the proposed approach. This work represents a significant step toward developing versatile, reasoning-enhanced AI systems for precision pathology. |
| title | A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model |
| topic | Image and Video Processing Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2507.17303 |