DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913753153404928 |
|---|---|
| author | Liang, Ruofan Gojcic, Zan Ling, Huan Munkberg, Jacob Hasselgren, Jon Lin, Zhi-Hao Gao, Jun Keller, Alexander Vijaykumar, Nandita Fidler, Sanja Wang, Zian |
| author_facet | Liang, Ruofan Gojcic, Zan Ling, Huan Munkberg, Jacob Hasselgren, Jon Lin, Zhi-Hao Gao, Jun Keller, Alexander Vijaykumar, Nandita Fidler, Sanja Wang, Zian |
| contents | Understanding and modeling lighting effects are fundamental tasks in computer vision and graphics. Classic physically-based rendering (PBR) accurately simulates the light transport, but relies on precise scene representations--explicit 3D geometry, high-quality material properties, and lighting conditions--that are often impractical to obtain in real-world scenarios. Therefore, we introduce DiffusionRenderer, a neural approach that addresses the dual problem of inverse and forward rendering within a holistic framework. Leveraging powerful video diffusion model priors, the inverse rendering model accurately estimates G-buffers from real-world videos, providing an interface for image editing tasks, and training data for the rendering model. Conversely, our rendering model generates photorealistic images from G-buffers without explicit light transport simulation. Experiments demonstrate that DiffusionRenderer effectively approximates inverse and forwards rendering, consistently outperforming the state-of-the-art. Our model enables practical applications from a single video input--including relighting, material editing, and realistic object insertion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_18590 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models Liang, Ruofan Gojcic, Zan Ling, Huan Munkberg, Jacob Hasselgren, Jon Lin, Zhi-Hao Gao, Jun Keller, Alexander Vijaykumar, Nandita Fidler, Sanja Wang, Zian Computer Vision and Pattern Recognition Graphics Understanding and modeling lighting effects are fundamental tasks in computer vision and graphics. Classic physically-based rendering (PBR) accurately simulates the light transport, but relies on precise scene representations--explicit 3D geometry, high-quality material properties, and lighting conditions--that are often impractical to obtain in real-world scenarios. Therefore, we introduce DiffusionRenderer, a neural approach that addresses the dual problem of inverse and forward rendering within a holistic framework. Leveraging powerful video diffusion model priors, the inverse rendering model accurately estimates G-buffers from real-world videos, providing an interface for image editing tasks, and training data for the rendering model. Conversely, our rendering model generates photorealistic images from G-buffers without explicit light transport simulation. Experiments demonstrate that DiffusionRenderer effectively approximates inverse and forwards rendering, consistently outperforming the state-of-the-art. Our model enables practical applications from a single video input--including relighting, material editing, and realistic object insertion. |
| title | DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models |
| topic | Computer Vision and Pattern Recognition Graphics |
| url | https://arxiv.org/abs/2501.18590 |