Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.01547 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917305592578048 |
|---|---|
| author | Yu, Jian Nguyen, Joakim Fang, Jinrui Naeem, Awais Cao, Zeyuan Krishnan, Sanjay Konz, Nicholas Chen, Tianlong Krishnan, Chandra Wang, Hairong Castillo, Edward Ding, Ying Shukla, Ankita |
| author_facet | Yu, Jian Nguyen, Joakim Fang, Jinrui Naeem, Awais Cao, Zeyuan Krishnan, Sanjay Konz, Nicholas Chen, Tianlong Krishnan, Chandra Wang, Hairong Castillo, Edward Ding, Ying Shukla, Ankita |
| contents | Accurate classification of pediatric central nervous system tumors remains challenging due to histological complexity and limited training data. While pathology foundation models have advanced whole-slide image (WSI) analysis, they often fail to leverage the rich, complementary information found in clinical text and tissue microarchitecture. To this end, we propose PathMoE, an interpretable multimodal framework that integrates H\&E slides, pathology reports, and nuclei-level cell graphs via an interaction-aware mixture-of-experts architecture built on state-of-the-art foundation models for each modality. By training specialized experts to capture modality uniqueness, redundancy, and synergy, PathMoE employs an input-dependent gating mechanism that dynamically weights these interactions, providing sample-level interpretability. We evaluate our framework on two dataset-specific classification tasks on an internal pediatric brain tumor dataset (PBT) and external TCGA datasets. PathMoE improves macro-F1 from 0.762 to 0.799 (+0.037) on PBT when integrating WSI, text, and graph modalities; on TCGA, augmenting WSI with graph knowledge improves macro-F1 from 0.668 to 0.709 (+0.041). These results demonstrate significant performance gains over state-of-the-art image-only baselines while revealing the specific modality interactions driving individual predictions. This interpretability is particularly critical for rare tumor subtypes, where transparent model reasoning is essential for clinical trust and diagnostic validation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_01547 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PathMoE: Interpretable Multimodal Interaction Experts for Pediatric Brain Tumor Classification Yu, Jian Nguyen, Joakim Fang, Jinrui Naeem, Awais Cao, Zeyuan Krishnan, Sanjay Konz, Nicholas Chen, Tianlong Krishnan, Chandra Wang, Hairong Castillo, Edward Ding, Ying Shukla, Ankita Computer Vision and Pattern Recognition Accurate classification of pediatric central nervous system tumors remains challenging due to histological complexity and limited training data. While pathology foundation models have advanced whole-slide image (WSI) analysis, they often fail to leverage the rich, complementary information found in clinical text and tissue microarchitecture. To this end, we propose PathMoE, an interpretable multimodal framework that integrates H\&E slides, pathology reports, and nuclei-level cell graphs via an interaction-aware mixture-of-experts architecture built on state-of-the-art foundation models for each modality. By training specialized experts to capture modality uniqueness, redundancy, and synergy, PathMoE employs an input-dependent gating mechanism that dynamically weights these interactions, providing sample-level interpretability. We evaluate our framework on two dataset-specific classification tasks on an internal pediatric brain tumor dataset (PBT) and external TCGA datasets. PathMoE improves macro-F1 from 0.762 to 0.799 (+0.037) on PBT when integrating WSI, text, and graph modalities; on TCGA, augmenting WSI with graph knowledge improves macro-F1 from 0.668 to 0.709 (+0.041). These results demonstrate significant performance gains over state-of-the-art image-only baselines while revealing the specific modality interactions driving individual predictions. This interpretability is particularly critical for rare tumor subtypes, where transparent model reasoning is essential for clinical trust and diagnostic validation. |
| title | PathMoE: Interpretable Multimodal Interaction Experts for Pediatric Brain Tumor Classification |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.01547 |