Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.02824 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910858670505984 |
|---|---|
| author | Oh, Yujin Seifert, Robert Cao, Yihan Clement, Christoph Ferdinandus, Justin Lapa, Constantin Liebich, Alessandro Amon, Michelle Enke, Johanna Song, Sifan Meng, Runqi Zeng, Fang Guo, Ning Li, Xiang Heidari, Pedram Rominger, Axel Shi, Kuangyu Li, Quanzheng |
| author_facet | Oh, Yujin Seifert, Robert Cao, Yihan Clement, Christoph Ferdinandus, Justin Lapa, Constantin Liebich, Alessandro Amon, Michelle Enke, Johanna Song, Sifan Meng, Runqi Zeng, Fang Guo, Ning Li, Xiang Heidari, Pedram Rominger, Axel Shi, Kuangyu Li, Quanzheng |
| contents | In oncology, Positron Emission Tomography-Computed Tomography (PET/CT) is widely used in cancer diagnosis, staging, and treatment monitoring, as it combines anatomical details from CT with functional metabolic activity and molecular marker expression information from PET. However, existing artificial intelligence-driven PET/CT analyses rely predominantly on task-specific models trained from scratch or on limited datasets, limiting their generalizability and robustness. To address this, we propose a foundation model approach specifically designed for multimodal PET/CT imaging. We introduce the Cross-Fraternal Twin Masked Autoencoder (FratMAE), a novel framework that effectively integrates whole-body anatomical and functional or molecular information. FratMAE employs separate Vision Transformer (ViT) encoders for PET and CT scans, along with cross-attention decoders that enable synergistic interactions between modalities during masked autoencoder training. Additionally, it incorporates textual metadata to enhance PET representation learning. By pre-training on PET/CT datasets, FratMAE captures intricate cross-modal relationships and global uptake patterns, achieving superior performance on downstream tasks and demonstrating its potential as a generalizable foundation model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_02824 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Developing a PET/CT Foundation Model for Cross-Modal Anatomical and Functional Imaging Oh, Yujin Seifert, Robert Cao, Yihan Clement, Christoph Ferdinandus, Justin Lapa, Constantin Liebich, Alessandro Amon, Michelle Enke, Johanna Song, Sifan Meng, Runqi Zeng, Fang Guo, Ning Li, Xiang Heidari, Pedram Rominger, Axel Shi, Kuangyu Li, Quanzheng Computer Vision and Pattern Recognition Artificial Intelligence In oncology, Positron Emission Tomography-Computed Tomography (PET/CT) is widely used in cancer diagnosis, staging, and treatment monitoring, as it combines anatomical details from CT with functional metabolic activity and molecular marker expression information from PET. However, existing artificial intelligence-driven PET/CT analyses rely predominantly on task-specific models trained from scratch or on limited datasets, limiting their generalizability and robustness. To address this, we propose a foundation model approach specifically designed for multimodal PET/CT imaging. We introduce the Cross-Fraternal Twin Masked Autoencoder (FratMAE), a novel framework that effectively integrates whole-body anatomical and functional or molecular information. FratMAE employs separate Vision Transformer (ViT) encoders for PET and CT scans, along with cross-attention decoders that enable synergistic interactions between modalities during masked autoencoder training. Additionally, it incorporates textual metadata to enhance PET representation learning. By pre-training on PET/CT datasets, FratMAE captures intricate cross-modal relationships and global uptake patterns, achieving superior performance on downstream tasks and demonstrating its potential as a generalizable foundation model. |
| title | Developing a PET/CT Foundation Model for Cross-Modal Anatomical and Functional Imaging |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2503.02824 |