Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915536221241344 |
|---|---|
| author | Dorent, Reuben Haouchine, Nazim Golby, Alexandra Frisken, Sarah Kapur, Tina Wells, William |
| author_facet | Dorent, Reuben Haouchine, Nazim Golby, Alexandra Frisken, Sarah Kapur, Tina Wells, William |
| contents | We propose a deep mixture of multimodal hierarchical variational auto-encoders called MMHVAE that synthesizes missing images from observed images in different modalities. MMHVAE's design focuses on tackling four challenges: (i) creating a complex latent representation of multimodal data to generate high-resolution images; (ii) encouraging the variational distributions to estimate the missing information needed for cross-modal image synthesis; (iii) learning to fuse multimodal information in the context of missing data; (iv) leveraging dataset-level information to handle incomplete data sets at training time. Extensive experiments are performed on the challenging problem of pre-operative brain multi-parametric magnetic resonance and intra-operative ultrasound imaging. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_19378 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts Dorent, Reuben Haouchine, Nazim Golby, Alexandra Frisken, Sarah Kapur, Tina Wells, William Computer Vision and Pattern Recognition Machine Learning Image and Video Processing We propose a deep mixture of multimodal hierarchical variational auto-encoders called MMHVAE that synthesizes missing images from observed images in different modalities. MMHVAE's design focuses on tackling four challenges: (i) creating a complex latent representation of multimodal data to generate high-resolution images; (ii) encouraging the variational distributions to estimate the missing information needed for cross-modal image synthesis; (iii) learning to fuse multimodal information in the context of missing data; (iv) leveraging dataset-level information to handle incomplete data sets at training time. Extensive experiments are performed on the challenging problem of pre-operative brain multi-parametric magnetic resonance and intra-operative ultrasound imaging. |
| title | Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts |
| topic | Computer Vision and Pattern Recognition Machine Learning Image and Video Processing |
| url | https://arxiv.org/abs/2410.19378 |