Reinforced Correlation Between Vision and Language for Precise Medical AI Assistant
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913823016878080 |
|---|---|
| author | Wang, Haonan Mao, Jiaji Wang, Lehan Zhang, Qixiang Elbatel, Marawan Qin, Yi Hu, Huijun Li, Baoxun Deng, Wenhui Qin, Weifeng Li, Hongrui Liang, Jialin Shen, Jun Li, Xiaomeng |
| author_facet | Wang, Haonan Mao, Jiaji Wang, Lehan Zhang, Qixiang Elbatel, Marawan Qin, Yi Hu, Huijun Li, Baoxun Deng, Wenhui Qin, Weifeng Li, Hongrui Liang, Jialin Shen, Jun Li, Xiaomeng |
| contents | Medical AI assistants support doctors in disease diagnosis, medical image analysis, and report generation. However, they still face significant challenges in clinical use, including limited accuracy with multimodal content and insufficient validation in real-world settings. We propose RCMed, a full-stack AI assistant that improves multimodal alignment in both input and output, enabling precise anatomical delineation, accurate localization, and reliable diagnosis through hierarchical vision-language grounding. A self-reinforcing correlation mechanism allows visual features to inform language context, while language semantics guide pixel-wise attention, forming a closed loop that refines both modalities. This correlation is enhanced by a color region description strategy, translating anatomical structures into semantically rich text to learn shape-location-text relationships across scales. Trained on 20 million image-mask-description triplets, RCMed achieves state-of-the-art precision in contextualizing irregular lesions and subtle anatomical boundaries, excelling in 165 clinical tasks across 9 modalities. It achieved a 23.5% relative improvement in cell segmentation from microscopy images over prior methods. RCMed's strong vision-language alignment enables exceptional generalization, with state-of-the-art performance in external validation across 20 clinically significant cancer types, including novel tasks. This work demonstrates how integrated multimodal models capture fine-grained patterns, enabling human-level interpretation in complex scenarios and advancing human-centric AI healthcare. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_03380 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reinforced Correlation Between Vision and Language for Precise Medical AI Assistant Wang, Haonan Mao, Jiaji Wang, Lehan Zhang, Qixiang Elbatel, Marawan Qin, Yi Hu, Huijun Li, Baoxun Deng, Wenhui Qin, Weifeng Li, Hongrui Liang, Jialin Shen, Jun Li, Xiaomeng Computer Vision and Pattern Recognition Artificial Intelligence Image and Video Processing Medical AI assistants support doctors in disease diagnosis, medical image analysis, and report generation. However, they still face significant challenges in clinical use, including limited accuracy with multimodal content and insufficient validation in real-world settings. We propose RCMed, a full-stack AI assistant that improves multimodal alignment in both input and output, enabling precise anatomical delineation, accurate localization, and reliable diagnosis through hierarchical vision-language grounding. A self-reinforcing correlation mechanism allows visual features to inform language context, while language semantics guide pixel-wise attention, forming a closed loop that refines both modalities. This correlation is enhanced by a color region description strategy, translating anatomical structures into semantically rich text to learn shape-location-text relationships across scales. Trained on 20 million image-mask-description triplets, RCMed achieves state-of-the-art precision in contextualizing irregular lesions and subtle anatomical boundaries, excelling in 165 clinical tasks across 9 modalities. It achieved a 23.5% relative improvement in cell segmentation from microscopy images over prior methods. RCMed's strong vision-language alignment enables exceptional generalization, with state-of-the-art performance in external validation across 20 clinically significant cancer types, including novel tasks. This work demonstrates how integrated multimodal models capture fine-grained patterns, enabling human-level interpretation in complex scenarios and advancing human-centric AI healthcare. |
| title | Reinforced Correlation Between Vision and Language for Precise Medical AI Assistant |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Image and Video Processing |
| url | https://arxiv.org/abs/2505.03380 |