Infinite-Resolution Integral Noise Warping for Diffusion Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912102730432512 |
|---|---|
| author | Deng, Yitong Lin, Winnie Li, Lingxiao Smirnov, Dmitriy Burgert, Ryan Yu, Ning Dedun, Vincent Taghavi, Mohammad H. |
| author_facet | Deng, Yitong Lin, Winnie Li, Lingxiao Smirnov, Dmitriy Burgert, Ryan Yu, Ning Dedun, Vincent Taghavi, Mohammad H. |
| contents | Adapting pretrained image-based diffusion models to generate temporally consistent videos has become an impactful generative modeling research direction. Training-free noise-space manipulation has proven to be an effective technique, where the challenge is to preserve the Gaussian white noise distribution while adding in temporal consistency. Recently, Chang et al. (2024) formulated this problem using an integral noise representation with distribution-preserving guarantees, and proposed an upsampling-based algorithm to compute it. However, while their mathematical formulation is advantageous, the algorithm incurs a high computational cost. Through analyzing the limiting-case behavior of their algorithm as the upsampling resolution goes to infinity, we develop an alternative algorithm that, by gathering increments of multiple Brownian bridges, achieves their infinite-resolution accuracy while simultaneously reducing the computational cost by orders of magnitude. We prove and experimentally validate our theoretical claims, and demonstrate our method's effectiveness in real-world applications. We further show that our method readily extends to the 3-dimensional space. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_01212 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Infinite-Resolution Integral Noise Warping for Diffusion Models Deng, Yitong Lin, Winnie Li, Lingxiao Smirnov, Dmitriy Burgert, Ryan Yu, Ning Dedun, Vincent Taghavi, Mohammad H. Computer Vision and Pattern Recognition Artificial Intelligence Graphics Machine Learning Adapting pretrained image-based diffusion models to generate temporally consistent videos has become an impactful generative modeling research direction. Training-free noise-space manipulation has proven to be an effective technique, where the challenge is to preserve the Gaussian white noise distribution while adding in temporal consistency. Recently, Chang et al. (2024) formulated this problem using an integral noise representation with distribution-preserving guarantees, and proposed an upsampling-based algorithm to compute it. However, while their mathematical formulation is advantageous, the algorithm incurs a high computational cost. Through analyzing the limiting-case behavior of their algorithm as the upsampling resolution goes to infinity, we develop an alternative algorithm that, by gathering increments of multiple Brownian bridges, achieves their infinite-resolution accuracy while simultaneously reducing the computational cost by orders of magnitude. We prove and experimentally validate our theoretical claims, and demonstrate our method's effectiveness in real-world applications. We further show that our method readily extends to the 3-dimensional space. |
| title | Infinite-Resolution Integral Noise Warping for Diffusion Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Graphics Machine Learning |
| url | https://arxiv.org/abs/2411.01212 |