Disentangled Multimodal Brain MR Image Translation via Transformer-based Modality Infuser
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917580272304128 |
|---|---|
| author | Cho, Jihoon Liu, Xiaofeng Xing, Fangxu Ouyang, Jinsong Fakhri, Georges El Park, Jinah Woo, Jonghye |
| author_facet | Cho, Jihoon Liu, Xiaofeng Xing, Fangxu Ouyang, Jinsong Fakhri, Georges El Park, Jinah Woo, Jonghye |
| contents | Multimodal Magnetic Resonance (MR) Imaging plays a crucial role in disease diagnosis due to its ability to provide complementary information by analyzing a relationship between multimodal images on the same subject. Acquiring all MR modalities, however, can be expensive, and, during a scanning session, certain MR images may be missed depending on the study protocol. The typical solution would be to synthesize the missing modalities from the acquired images such as using generative adversarial networks (GANs). Yet, GANs constructed with convolutional neural networks (CNNs) are likely to suffer from a lack of global relationships and mechanisms to condition the desired modality. To address this, in this work, we propose a transformer-based modality infuser designed to synthesize multimodal brain MR images. In our method, we extract modality-agnostic features from the encoder and then transform them into modality-specific features using the modality infuser. Furthermore, the modality infuser captures long-range relationships among all brain structures, leading to the generation of more realistic images. We carried out experiments on the BraTS 2018 dataset, translating between four MR modalities, and our experimental results demonstrate the superiority of our proposed method in terms of synthesis quality. In addition, we conducted experiments on a brain tumor segmentation task and different conditioning methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_00375 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Disentangled Multimodal Brain MR Image Translation via Transformer-based Modality Infuser Cho, Jihoon Liu, Xiaofeng Xing, Fangxu Ouyang, Jinsong Fakhri, Georges El Park, Jinah Woo, Jonghye Image and Video Processing Computer Vision and Pattern Recognition Multimodal Magnetic Resonance (MR) Imaging plays a crucial role in disease diagnosis due to its ability to provide complementary information by analyzing a relationship between multimodal images on the same subject. Acquiring all MR modalities, however, can be expensive, and, during a scanning session, certain MR images may be missed depending on the study protocol. The typical solution would be to synthesize the missing modalities from the acquired images such as using generative adversarial networks (GANs). Yet, GANs constructed with convolutional neural networks (CNNs) are likely to suffer from a lack of global relationships and mechanisms to condition the desired modality. To address this, in this work, we propose a transformer-based modality infuser designed to synthesize multimodal brain MR images. In our method, we extract modality-agnostic features from the encoder and then transform them into modality-specific features using the modality infuser. Furthermore, the modality infuser captures long-range relationships among all brain structures, leading to the generation of more realistic images. We carried out experiments on the BraTS 2018 dataset, translating between four MR modalities, and our experimental results demonstrate the superiority of our proposed method in terms of synthesis quality. In addition, we conducted experiments on a brain tumor segmentation task and different conditioning methods. |
| title | Disentangled Multimodal Brain MR Image Translation via Transformer-based Modality Infuser |
| topic | Image and Video Processing Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.00375 |