IMPUS: Image Morphing with Perceptually-Uniform Sampling Using Diffusion Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916160702775296 |
|---|---|
| author | Yang, Zhaoyuan Yu, Zhengyang Xu, Zhiwei Singh, Jaskirat Zhang, Jing Campbell, Dylan Tu, Peter Hartley, Richard |
| author_facet | Yang, Zhaoyuan Yu, Zhengyang Xu, Zhiwei Singh, Jaskirat Zhang, Jing Campbell, Dylan Tu, Peter Hartley, Richard |
| contents | We present a diffusion-based image morphing approach with perceptually-uniform sampling (IMPUS) that produces smooth, direct and realistic interpolations given an image pair. The embeddings of two images may lie on distinct conditioned distributions of a latent diffusion model, especially when they have significant semantic difference. To bridge this gap, we interpolate in the locally linear and continuous text embedding space and Gaussian latent space. We first optimize the endpoint text embeddings and then map the images to the latent space using a probability flow ODE. Unlike existing work that takes an indirect morphing path, we show that the model adaptation yields a direct path and suppresses ghosting artifacts in the interpolated images. To achieve this, we propose a heuristic bottleneck constraint based on a novel relative perceptual path diversity score that automatically controls the bottleneck size and balances the diversity along the path with its directness. We also propose a perceptually-uniform sampling technique that enables visually smooth changes between the interpolated images. Extensive experiments validate that our IMPUS can achieve smooth, direct, and realistic image morphing and is adaptable to several other generative tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_06792 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | IMPUS: Image Morphing with Perceptually-Uniform Sampling Using Diffusion Models Yang, Zhaoyuan Yu, Zhengyang Xu, Zhiwei Singh, Jaskirat Zhang, Jing Campbell, Dylan Tu, Peter Hartley, Richard Computer Vision and Pattern Recognition We present a diffusion-based image morphing approach with perceptually-uniform sampling (IMPUS) that produces smooth, direct and realistic interpolations given an image pair. The embeddings of two images may lie on distinct conditioned distributions of a latent diffusion model, especially when they have significant semantic difference. To bridge this gap, we interpolate in the locally linear and continuous text embedding space and Gaussian latent space. We first optimize the endpoint text embeddings and then map the images to the latent space using a probability flow ODE. Unlike existing work that takes an indirect morphing path, we show that the model adaptation yields a direct path and suppresses ghosting artifacts in the interpolated images. To achieve this, we propose a heuristic bottleneck constraint based on a novel relative perceptual path diversity score that automatically controls the bottleneck size and balances the diversity along the path with its directness. We also propose a perceptually-uniform sampling technique that enables visually smooth changes between the interpolated images. Extensive experiments validate that our IMPUS can achieve smooth, direct, and realistic image morphing and is adaptable to several other generative tasks. |
| title | IMPUS: Image Morphing with Perceptually-Uniform Sampling Using Diffusion Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2311.06792 |