Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.10059 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915384832032768 |
|---|---|
| author | Li, Teng Zheng, Guangcong Jiang, Rui Zhan, Shuigen Wu, Tao Lu, Yehao Lin, Yining Deng, Chuanyun Xiong, Yepan Chen, Min Cheng, Lin Li, Xi |
| author_facet | Li, Teng Zheng, Guangcong Jiang, Rui Zhan, Shuigen Wu, Tao Lu, Yehao Lin, Yining Deng, Chuanyun Xiong, Yepan Chen, Min Cheng, Lin Li, Xi |
| contents | Recent advancements in camera-trajectory-guided image-to-video generation offer higher precision and better support for complex camera control compared to text-based approaches. However, they also introduce significant usability challenges, as users often struggle to provide precise camera parameters when working with arbitrary real-world images without knowledge of their depth nor scene scale. To address these real-world application issues, we propose RealCam-I2V, a novel diffusion-based video generation framework that integrates monocular metric depth estimation to establish 3D scene reconstruction in a preprocessing step. During training, the reconstructed 3D scene enables scaling camera parameters from relative to metric scales, ensuring compatibility and scale consistency across diverse real-world images. In inference, RealCam-I2V offers an intuitive interface where users can precisely draw camera trajectories by dragging within the 3D scene. To further enhance precise camera control and scene consistency, we propose scene-constrained noise shaping, which shapes high-level noise and also allows the framework to maintain dynamic and coherent video generation in lower noise stages. RealCam-I2V achieves significant improvements in controllability and video quality on the RealEstate10K and out-of-domain images. We further enables applications like camera-controlled looping video generation and generative frame interpolation. Project page: https://zgctroy.github.io/RealCam-I2V. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_10059 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Control Li, Teng Zheng, Guangcong Jiang, Rui Zhan, Shuigen Wu, Tao Lu, Yehao Lin, Yining Deng, Chuanyun Xiong, Yepan Chen, Min Cheng, Lin Li, Xi Computer Vision and Pattern Recognition Recent advancements in camera-trajectory-guided image-to-video generation offer higher precision and better support for complex camera control compared to text-based approaches. However, they also introduce significant usability challenges, as users often struggle to provide precise camera parameters when working with arbitrary real-world images without knowledge of their depth nor scene scale. To address these real-world application issues, we propose RealCam-I2V, a novel diffusion-based video generation framework that integrates monocular metric depth estimation to establish 3D scene reconstruction in a preprocessing step. During training, the reconstructed 3D scene enables scaling camera parameters from relative to metric scales, ensuring compatibility and scale consistency across diverse real-world images. In inference, RealCam-I2V offers an intuitive interface where users can precisely draw camera trajectories by dragging within the 3D scene. To further enhance precise camera control and scene consistency, we propose scene-constrained noise shaping, which shapes high-level noise and also allows the framework to maintain dynamic and coherent video generation in lower noise stages. RealCam-I2V achieves significant improvements in controllability and video quality on the RealEstate10K and out-of-domain images. We further enables applications like camera-controlled looping video generation and generative frame interpolation. Project page: https://zgctroy.github.io/RealCam-I2V. |
| title | RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Control |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.10059 |