Global Contrastive Training for Multimodal Electronic Health Records with Language Supervision
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914747734032384 |
|---|---|
| author | Ma, Yingbo Kolla, Suraj Hu, Zhenhong Kaliraman, Dhruv Nolan, Victoria Guan, Ziyuan Ren, Yuanfang Armfield, Brooke Ozrazgat-Baslanti, Tezcan Balch, Jeremy A. Loftus, Tyler J. Rashidi, Parisa Bihorac, Azra Shickel, Benjamin |
| author_facet | Ma, Yingbo Kolla, Suraj Hu, Zhenhong Kaliraman, Dhruv Nolan, Victoria Guan, Ziyuan Ren, Yuanfang Armfield, Brooke Ozrazgat-Baslanti, Tezcan Balch, Jeremy A. Loftus, Tyler J. Rashidi, Parisa Bihorac, Azra Shickel, Benjamin |
| contents | Modern electronic health records (EHRs) hold immense promise in tracking personalized patient health trajectories through sequential deep learning, owing to their extensive breadth, scale, and temporal granularity. Nonetheless, how to effectively leverage multiple modalities from EHRs poses significant challenges, given its complex characteristics such as high dimensionality, multimodality, sparsity, varied recording frequencies, and temporal irregularities. To this end, this paper introduces a novel multimodal contrastive learning framework, specifically focusing on medical time series and clinical notes. To tackle the challenge of sparsity and irregular time intervals in medical time series, the framework integrates temporal cross-attention transformers with a dynamic embedding and tokenization scheme for learning multimodal feature representations. To harness the interconnected relationships between medical time series and clinical notes, the framework equips a global contrastive loss, aligning a patient's multimodal feature representations with the corresponding discharge summaries. Since discharge summaries uniquely pertain to individual patients and represent a holistic view of the patient's hospital stay, machine learning models are led to learn discriminative multimodal features via global contrasting. Extensive experiments with a real-world EHR dataset demonstrated that our framework outperformed state-of-the-art approaches on the exemplar task of predicting the occurrence of nine postoperative complications for more than 120,000 major inpatient surgeries using multimodal data from UF health system split among three hospitals (UF Health Gainesville, UF Health Jacksonville, and UF Health Jacksonville-North). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_06723 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Global Contrastive Training for Multimodal Electronic Health Records with Language Supervision Ma, Yingbo Kolla, Suraj Hu, Zhenhong Kaliraman, Dhruv Nolan, Victoria Guan, Ziyuan Ren, Yuanfang Armfield, Brooke Ozrazgat-Baslanti, Tezcan Balch, Jeremy A. Loftus, Tyler J. Rashidi, Parisa Bihorac, Azra Shickel, Benjamin Machine Learning Computation and Language Modern electronic health records (EHRs) hold immense promise in tracking personalized patient health trajectories through sequential deep learning, owing to their extensive breadth, scale, and temporal granularity. Nonetheless, how to effectively leverage multiple modalities from EHRs poses significant challenges, given its complex characteristics such as high dimensionality, multimodality, sparsity, varied recording frequencies, and temporal irregularities. To this end, this paper introduces a novel multimodal contrastive learning framework, specifically focusing on medical time series and clinical notes. To tackle the challenge of sparsity and irregular time intervals in medical time series, the framework integrates temporal cross-attention transformers with a dynamic embedding and tokenization scheme for learning multimodal feature representations. To harness the interconnected relationships between medical time series and clinical notes, the framework equips a global contrastive loss, aligning a patient's multimodal feature representations with the corresponding discharge summaries. Since discharge summaries uniquely pertain to individual patients and represent a holistic view of the patient's hospital stay, machine learning models are led to learn discriminative multimodal features via global contrasting. Extensive experiments with a real-world EHR dataset demonstrated that our framework outperformed state-of-the-art approaches on the exemplar task of predicting the occurrence of nine postoperative complications for more than 120,000 major inpatient surgeries using multimodal data from UF health system split among three hospitals (UF Health Gainesville, UF Health Jacksonville, and UF Health Jacksonville-North). |
| title | Global Contrastive Training for Multimodal Electronic Health Records with Language Supervision |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2404.06723 |