STeP: A Framework for Solving Scientific Video Inverse Problems with Spatiotemporal Diffusion Priors
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912421870829568 |
|---|---|
| author | Zhang, Bingliang Wu, Zihui Feng, Berthy T. Song, Yang Yue, Yisong Bouman, Katherine L. |
| author_facet | Zhang, Bingliang Wu, Zihui Feng, Berthy T. Song, Yang Yue, Yisong Bouman, Katherine L. |
| contents | Reconstructing spatially and temporally coherent videos from time-varying measurements is a fundamental challenge in many scientific domains. A major difficulty arises from the sparsity of measurements, which hinders accurate recovery of temporal dynamics. Existing image diffusion-based methods rely on extracting temporal consistency directly from measurements, limiting their effectiveness on scientific tasks with high spatiotemporal uncertainty. We address this difficulty by proposing a plug-and-play framework that incorporates a learned spatiotemporal diffusion prior. Due to its plug-and-play nature, our framework can be flexibly applied to different video inverse problems without the need for task-specific design and temporal heuristics. We further demonstrate that a spatiotemporal diffusion model can be trained efficiently with limited video data. We validate our approach on two challenging scientific video reconstruction tasks: black hole video reconstruction and dynamic MRI. While baseline methods struggle to provide temporally coherent reconstructions, our approach achieves significantly improved recovery of the spatiotemporal structure of the underlying ground truth videos. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_07549 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | STeP: A Framework for Solving Scientific Video Inverse Problems with Spatiotemporal Diffusion Priors Zhang, Bingliang Wu, Zihui Feng, Berthy T. Song, Yang Yue, Yisong Bouman, Katherine L. Computer Vision and Pattern Recognition Reconstructing spatially and temporally coherent videos from time-varying measurements is a fundamental challenge in many scientific domains. A major difficulty arises from the sparsity of measurements, which hinders accurate recovery of temporal dynamics. Existing image diffusion-based methods rely on extracting temporal consistency directly from measurements, limiting their effectiveness on scientific tasks with high spatiotemporal uncertainty. We address this difficulty by proposing a plug-and-play framework that incorporates a learned spatiotemporal diffusion prior. Due to its plug-and-play nature, our framework can be flexibly applied to different video inverse problems without the need for task-specific design and temporal heuristics. We further demonstrate that a spatiotemporal diffusion model can be trained efficiently with limited video data. We validate our approach on two challenging scientific video reconstruction tasks: black hole video reconstruction and dynamic MRI. While baseline methods struggle to provide temporally coherent reconstructions, our approach achieves significantly improved recovery of the spatiotemporal structure of the underlying ground truth videos. |
| title | STeP: A Framework for Solving Scientific Video Inverse Problems with Spatiotemporal Diffusion Priors |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2504.07549 |