SpotLight: Shadow-Guided Object Relighting via Diffusion
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911311786409984 |
|---|---|
| author | Fortier-Chouinard, Frédéric Zhang, Zitian Messier, Louis-Etienne Garon, Mathieu Bhattad, Anand Lalonde, Jean-François |
| author_facet | Fortier-Chouinard, Frédéric Zhang, Zitian Messier, Louis-Etienne Garon, Mathieu Bhattad, Anand Lalonde, Jean-François |
| contents | Recent work has shown that diffusion models can serve as powerful neural rendering engines that can be leveraged for inserting virtual objects into images. However, unlike typical physics-based renderers, these neural rendering engines are limited by the lack of manual control over the lighting, which is often essential for improving or personalizing the desired image outcome. In this paper, we show that precise and controllable lighting can be achieved without any additional training, simply by supplying a coarse shadow hint for the object. Indeed, we show that injecting only the desired shadow of the object into a pre-trained diffusion-based neural renderer enables it to accurately shade the object according to the desired light position, while properly harmonizing the object (and its shadow) within the target background image. Our method, SpotLight, is entirely training-free and leverages existing neural rendering approaches to achieve controllable relighting. We show that SpotLight achieves superior object compositing results, both quantitatively and perceptually, as confirmed by a user study, outperforming existing diffusion-based models specifically designed for relighting. We also demonstrate other applications, such as hand-scribbling shadows and full-image relighting, demonstrating its versatility. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_18665 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SpotLight: Shadow-Guided Object Relighting via Diffusion Fortier-Chouinard, Frédéric Zhang, Zitian Messier, Louis-Etienne Garon, Mathieu Bhattad, Anand Lalonde, Jean-François Computer Vision and Pattern Recognition Graphics Recent work has shown that diffusion models can serve as powerful neural rendering engines that can be leveraged for inserting virtual objects into images. However, unlike typical physics-based renderers, these neural rendering engines are limited by the lack of manual control over the lighting, which is often essential for improving or personalizing the desired image outcome. In this paper, we show that precise and controllable lighting can be achieved without any additional training, simply by supplying a coarse shadow hint for the object. Indeed, we show that injecting only the desired shadow of the object into a pre-trained diffusion-based neural renderer enables it to accurately shade the object according to the desired light position, while properly harmonizing the object (and its shadow) within the target background image. Our method, SpotLight, is entirely training-free and leverages existing neural rendering approaches to achieve controllable relighting. We show that SpotLight achieves superior object compositing results, both quantitatively and perceptually, as confirmed by a user study, outperforming existing diffusion-based models specifically designed for relighting. We also demonstrate other applications, such as hand-scribbling shadows and full-image relighting, demonstrating its versatility. |
| title | SpotLight: Shadow-Guided Object Relighting via Diffusion |
| topic | Computer Vision and Pattern Recognition Graphics |
| url | https://arxiv.org/abs/2411.18665 |