Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2508.05492 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913979209613312 |
|---|---|
| author | Gao, Jifan Rahman, Mahmudur Caskey, John Oguss, Madeline O'Rourke, Ann Brown, Randy Stey, Anne Mayampurath, Anoop Churpek, Matthew M. Chen, Guanhua Afshar, Majid |
| author_facet | Gao, Jifan Rahman, Mahmudur Caskey, John Oguss, Madeline O'Rourke, Ann Brown, Randy Stey, Anne Mayampurath, Anoop Churpek, Matthew M. Chen, Guanhua Afshar, Majid |
| contents | Multimodal electronic health record (EHR) data provide richer, complementary insights into patient health compared to single-modality data. However, effectively integrating diverse data modalities for clinical prediction modeling remains challenging due to the substantial data requirements. We introduce a novel architecture, Mixture-of-Multimodal-Agents (MoMA), designed to leverage multiple large language model (LLM) agents for clinical prediction tasks using multimodal EHR data. MoMA employs specialized LLM agents ("specialist agents") to convert non-textual modalities, such as medical images and laboratory results, into structured textual summaries. These summaries, together with clinical notes, are combined by another LLM ("aggregator agent") to generate a unified multimodal summary, which is then used by a third LLM ("predictor agent") to produce clinical predictions. Evaluating MoMA on three prediction tasks using real-world datasets with different modality combinations and prediction settings, MoMA outperforms current state-of-the-art methods, highlighting its enhanced accuracy and flexibility across various tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_05492 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modelling Gao, Jifan Rahman, Mahmudur Caskey, John Oguss, Madeline O'Rourke, Ann Brown, Randy Stey, Anne Mayampurath, Anoop Churpek, Matthew M. Chen, Guanhua Afshar, Majid Machine Learning Artificial Intelligence Multiagent Systems Multimodal electronic health record (EHR) data provide richer, complementary insights into patient health compared to single-modality data. However, effectively integrating diverse data modalities for clinical prediction modeling remains challenging due to the substantial data requirements. We introduce a novel architecture, Mixture-of-Multimodal-Agents (MoMA), designed to leverage multiple large language model (LLM) agents for clinical prediction tasks using multimodal EHR data. MoMA employs specialized LLM agents ("specialist agents") to convert non-textual modalities, such as medical images and laboratory results, into structured textual summaries. These summaries, together with clinical notes, are combined by another LLM ("aggregator agent") to generate a unified multimodal summary, which is then used by a third LLM ("predictor agent") to produce clinical predictions. Evaluating MoMA on three prediction tasks using real-world datasets with different modality combinations and prediction settings, MoMA outperforms current state-of-the-art methods, highlighting its enhanced accuracy and flexibility across various tasks. |
| title | MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modelling |
| topic | Machine Learning Artificial Intelligence Multiagent Systems |
| url | https://arxiv.org/abs/2508.05492 |