Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.03150 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918044690808832 |
|---|---|
| author | Lin, Yuanze Chen, Yi-Wen Tsai, Yi-Hsuan Clark, Ronald Yang, Ming-Hsuan |
| author_facet | Lin, Yuanze Chen, Yi-Wen Tsai, Yi-Hsuan Clark, Ronald Yang, Ming-Hsuan |
| contents | Although diffusion-based models can generate high-quality and high-resolution video sequences from textual or image inputs, they lack explicit integration of geometric cues when controlling scene lighting and visual appearance across frames. To address this limitation, we propose IllumiCraft, an end-to-end diffusion framework accepting three complementary inputs: (1) high-dynamic-range (HDR) video maps for detailed lighting control; (2) synthetically relit frames with randomized illumination changes (optionally paired with a static background reference image) to provide appearance cues; and (3) 3D point tracks that capture precise 3D geometry information. By integrating the lighting, appearance, and geometry cues within a unified diffusion architecture, IllumiCraft generates temporally coherent videos aligned with user-defined prompts. It supports background-conditioned and text-conditioned video relighting and provides better fidelity than existing controllable video generation methods. Project Page: https://yuanze-lin.me/IllumiCraft_page |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_03150 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation Lin, Yuanze Chen, Yi-Wen Tsai, Yi-Hsuan Clark, Ronald Yang, Ming-Hsuan Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Multimedia Although diffusion-based models can generate high-quality and high-resolution video sequences from textual or image inputs, they lack explicit integration of geometric cues when controlling scene lighting and visual appearance across frames. To address this limitation, we propose IllumiCraft, an end-to-end diffusion framework accepting three complementary inputs: (1) high-dynamic-range (HDR) video maps for detailed lighting control; (2) synthetically relit frames with randomized illumination changes (optionally paired with a static background reference image) to provide appearance cues; and (3) 3D point tracks that capture precise 3D geometry information. By integrating the lighting, appearance, and geometry cues within a unified diffusion architecture, IllumiCraft generates temporally coherent videos aligned with user-defined prompts. It supports background-conditioned and text-conditioned video relighting and provides better fidelity than existing controllable video generation methods. Project Page: https://yuanze-lin.me/IllumiCraft_page |
| title | IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Multimedia |
| url | https://arxiv.org/abs/2506.03150 |