Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.05143 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909338848722944 |
|---|---|
| author | Efimov, Timofey Dong, Harry Shah, Megna Simmons, Jeff Donegan, Sean Chi, Yuejie |
| author_facet | Efimov, Timofey Dong, Harry Shah, Megna Simmons, Jeff Donegan, Sean Chi, Yuejie |
| contents | Diffusion models have found phenomenal success as expressive priors for solving inverse problems, but their extension beyond natural images to more structured scientific domains remains limited. Motivated by applications in materials science, we aim to reduce the number of measurements required from an expensive imaging modality of interest, by leveraging side information from an auxiliary modality that is much cheaper to obtain. To deal with the non-differentiable and black-box nature of the forward model, we propose a framework to train a multimodal diffusion model over the joint modalities, turning inverse problems with black-box forward models into simple linear inpainting problems. Numerically, we demonstrate the feasibility of training diffusion models over materials imagery data, and show that our approach achieves superior image reconstruction by leveraging the available side information, requiring significantly less amount of data from the expensive microscopy modality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_05143 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Leveraging Multimodal Diffusion Models to Accelerate Imaging with Side Information Efimov, Timofey Dong, Harry Shah, Megna Simmons, Jeff Donegan, Sean Chi, Yuejie Computer Vision and Pattern Recognition Diffusion models have found phenomenal success as expressive priors for solving inverse problems, but their extension beyond natural images to more structured scientific domains remains limited. Motivated by applications in materials science, we aim to reduce the number of measurements required from an expensive imaging modality of interest, by leveraging side information from an auxiliary modality that is much cheaper to obtain. To deal with the non-differentiable and black-box nature of the forward model, we propose a framework to train a multimodal diffusion model over the joint modalities, turning inverse problems with black-box forward models into simple linear inpainting problems. Numerically, we demonstrate the feasibility of training diffusion models over materials imagery data, and show that our approach achieves superior image reconstruction by leveraging the available side information, requiring significantly less amount of data from the expensive microscopy modality. |
| title | Leveraging Multimodal Diffusion Models to Accelerate Imaging with Side Information |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.05143 |