HeartcareGPT: A Unified Multimodal ECG Suite for Dual Signal-Image Modeling and Understanding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911570672484352 |
|---|---|
| author | Xie, Yihan Li, Sijing Lin, Tianwei Wang, Zhuonan Yang, Chenglin Zhong, Yu Yan, Wenjie Zhang, Wenqiao Guo, Xiaogang Xiao, Jun Zhuang, Yueting Ooi, Beng Chin |
| author_facet | Xie, Yihan Li, Sijing Lin, Tianwei Wang, Zhuonan Yang, Chenglin Zhong, Yu Yan, Wenjie Zhang, Wenqiao Guo, Xiaogang Xiao, Jun Zhuang, Yueting Ooi, Beng Chin |
| contents | Although electrocardiograms (ECG) play a dominant role in cardiovascular diagnosis and treatment, their intrinsic data forms and representational patterns pose significant challenges for medical multimodal large language models (Med-MLLMs) in achieving cross-modal semantic alignment. To address this gap, we propose Heartcare Suite, a unified ECG suite designed for dual signal-image modeling and understanding: (i) Heartcare-400K. A fine-grained ECG instruction dataset on top of our data pipeline engine--HeartAgent--by integrating high quality clinical ECG reports from top hospitals with open-source data. (ii) Heartcare-Bench. A systematic benchmark assessing performance of models in multi-perspective ECG understanding and cross-modal generalization, providing guidance for optimizing ECG comprehension models. (iii) HeartcareGPT. Built upon a structure-aware discrete tokenizer Beat, we propose Dual Stream Projection Alignment (DSPA) paradigm--a dual encoder projection alignment mechanism enabling joint optimizing and modeling native ECG signal-image within a shared feature space. HeartcareGPT achieves consistent improvements across diverse ECG understanding tasks, validating both the effectiveness of the unified modeling paradigm and the necessity of a high-quality data pipeline, and establishing a methodological foundation for extending Med-MLLMs towards physiological signal domains. Our project is available at https://github.com/ZJU4HealthCare/HeartcareGPT . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_05831 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HeartcareGPT: A Unified Multimodal ECG Suite for Dual Signal-Image Modeling and Understanding Xie, Yihan Li, Sijing Lin, Tianwei Wang, Zhuonan Yang, Chenglin Zhong, Yu Yan, Wenjie Zhang, Wenqiao Guo, Xiaogang Xiao, Jun Zhuang, Yueting Ooi, Beng Chin Machine Learning Artificial Intelligence Although electrocardiograms (ECG) play a dominant role in cardiovascular diagnosis and treatment, their intrinsic data forms and representational patterns pose significant challenges for medical multimodal large language models (Med-MLLMs) in achieving cross-modal semantic alignment. To address this gap, we propose Heartcare Suite, a unified ECG suite designed for dual signal-image modeling and understanding: (i) Heartcare-400K. A fine-grained ECG instruction dataset on top of our data pipeline engine--HeartAgent--by integrating high quality clinical ECG reports from top hospitals with open-source data. (ii) Heartcare-Bench. A systematic benchmark assessing performance of models in multi-perspective ECG understanding and cross-modal generalization, providing guidance for optimizing ECG comprehension models. (iii) HeartcareGPT. Built upon a structure-aware discrete tokenizer Beat, we propose Dual Stream Projection Alignment (DSPA) paradigm--a dual encoder projection alignment mechanism enabling joint optimizing and modeling native ECG signal-image within a shared feature space. HeartcareGPT achieves consistent improvements across diverse ECG understanding tasks, validating both the effectiveness of the unified modeling paradigm and the necessity of a high-quality data pipeline, and establishing a methodological foundation for extending Med-MLLMs towards physiological signal domains. Our project is available at https://github.com/ZJU4HealthCare/HeartcareGPT . |
| title | HeartcareGPT: A Unified Multimodal ECG Suite for Dual Signal-Image Modeling and Understanding |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2506.05831 |