Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.00574 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915367781138432 |
|---|---|
| author | Rajamohan, Haresh Rengaraj Gao, Xiang Zhu, Weicheng Huang, Shih-Lun Chen, Long Cho, Kyunghyun Deniz, Cem M. Razavian, Narges |
| author_facet | Rajamohan, Haresh Rengaraj Gao, Xiang Zhu, Weicheng Huang, Shih-Lun Chen, Long Cho, Kyunghyun Deniz, Cem M. Razavian, Narges |
| contents | Large-scale pretraining has transformed modeling of language and other data types, but its potential remains underexplored in healthcare with structured electronic health records (EHRs). We present a novel generative pretraining strategy for sequential EHR data using next-visit event prediction. Our model learns to autoregressively generate various tokenized clinical events for the next visit based on patient history and inherently handles the joint prediction of heterogeneous data types. Additionally, we introduce regularization on predicting repeated events and highlight a key pitfall in EHR-based foundation model evaluations: repeated event tokens can inflate performance metrics when new onsets are not distinguished from subsequent occurrences. Our model is evaluated via zero-shot prediction for forecasting dementia and knee osteoarthritis incidence within 2 and 5 years, and the model performance rivals a fully fine-tuned masked pretrained Transformer baseline, demonstrating that our approach captures complex clinical dependencies without requiring costly task-specific fine-tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_00574 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Foundation Models for Clinical Records at Health System Scale Rajamohan, Haresh Rengaraj Gao, Xiang Zhu, Weicheng Huang, Shih-Lun Chen, Long Cho, Kyunghyun Deniz, Cem M. Razavian, Narges Machine Learning Large-scale pretraining has transformed modeling of language and other data types, but its potential remains underexplored in healthcare with structured electronic health records (EHRs). We present a novel generative pretraining strategy for sequential EHR data using next-visit event prediction. Our model learns to autoregressively generate various tokenized clinical events for the next visit based on patient history and inherently handles the joint prediction of heterogeneous data types. Additionally, we introduce regularization on predicting repeated events and highlight a key pitfall in EHR-based foundation model evaluations: repeated event tokens can inflate performance metrics when new onsets are not distinguished from subsequent occurrences. Our model is evaluated via zero-shot prediction for forecasting dementia and knee osteoarthritis incidence within 2 and 5 years, and the model performance rivals a fully fine-tuned masked pretrained Transformer baseline, demonstrating that our approach captures complex clinical dependencies without requiring costly task-specific fine-tuning. |
| title | Foundation Models for Clinical Records at Health System Scale |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2507.00574 |