Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.18512 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915357275455488 |
|---|---|
| author | Zhang, Yuting Yuan, Kaishen Lu, Hao Yue, Yutao Chen, Jintai Wu, Kaishun |
| author_facet | Zhang, Yuting Yuan, Kaishen Lu, Hao Yue, Yutao Chen, Jintai Wu, Kaishun |
| contents | Accurate and interpretable multi-disease diagnosis remains a critical challenge in medical research, particularly when leveraging heterogeneous multimodal medical data. Current approaches often rely on single-modal data, limiting their ability to comprehensively understand complex diseases. To address this, we propose MedTVT-R1, a novel Multimodal Large Language Model (MLLM) framework designed to integrate clinical multimodal data for reasoning and diagnosing multiple diseases. We construct MedTVT-QA, a curated instruction dataset that provides question-answer pairs for physiological-level interpretations and disease-level diagnoses with a Chain of Evidence approach. MedTVT-R1 incorporates a modality perception layer to capture inter-modal dependencies and adaptively weight modality contributions. Additionally, we employ Group Relative Policy Optimization (GRPO)-based Reinforcement Fine-Tuning with a Jaccard Reward function to enhance diagnostic reasoning. Experimental results demonstrate MedTVT-R1's superiority in multimodal feature utilization and multi-disease diagnosis, offering significant potential for clinical applications such as diagnostic report generation and comorbidity reasoning. The dataset and code are available at https://github.com/keke-nice/MedTVT-R1. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_18512 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosis Zhang, Yuting Yuan, Kaishen Lu, Hao Yue, Yutao Chen, Jintai Wu, Kaishun Image and Video Processing Computation and Language Computer Vision and Pattern Recognition Quantitative Methods Accurate and interpretable multi-disease diagnosis remains a critical challenge in medical research, particularly when leveraging heterogeneous multimodal medical data. Current approaches often rely on single-modal data, limiting their ability to comprehensively understand complex diseases. To address this, we propose MedTVT-R1, a novel Multimodal Large Language Model (MLLM) framework designed to integrate clinical multimodal data for reasoning and diagnosing multiple diseases. We construct MedTVT-QA, a curated instruction dataset that provides question-answer pairs for physiological-level interpretations and disease-level diagnoses with a Chain of Evidence approach. MedTVT-R1 incorporates a modality perception layer to capture inter-modal dependencies and adaptively weight modality contributions. Additionally, we employ Group Relative Policy Optimization (GRPO)-based Reinforcement Fine-Tuning with a Jaccard Reward function to enhance diagnostic reasoning. Experimental results demonstrate MedTVT-R1's superiority in multimodal feature utilization and multi-disease diagnosis, offering significant potential for clinical applications such as diagnostic report generation and comorbidity reasoning. The dataset and code are available at https://github.com/keke-nice/MedTVT-R1. |
| title | MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosis |
| topic | Image and Video Processing Computation and Language Computer Vision and Pattern Recognition Quantitative Methods |
| url | https://arxiv.org/abs/2506.18512 |