Thinking Outside the BBox: Unconstrained Generative Object Compositing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916388538417152 |
|---|---|
| author | Tarrés, Gemma Canet Lin, Zhe Zhang, Zhifei Zhang, Jianming Song, Yizhi Ruta, Dan Gilbert, Andrew Collomosse, John Kim, Soo Ye |
| author_facet | Tarrés, Gemma Canet Lin, Zhe Zhang, Zhifei Zhang, Jianming Song, Yizhi Ruta, Dan Gilbert, Andrew Collomosse, John Kim, Soo Ye |
| contents | Compositing an object into an image involves multiple non-trivial sub-tasks such as object placement and scaling, color/lighting harmonization, viewpoint/geometry adjustment, and shadow/reflection generation. Recent generative image compositing methods leverage diffusion models to handle multiple sub-tasks at once. However, existing models face limitations due to their reliance on masking the original object during training, which constrains their generation to the input mask. Furthermore, obtaining an accurate input mask specifying the location and scale of the object in a new image can be highly challenging. To overcome such limitations, we define a novel problem of unconstrained generative object compositing, i.e., the generation is not bounded by the mask, and train a diffusion-based model on a synthesized paired dataset. Our first-of-its-kind model is able to generate object effects such as shadows and reflections that go beyond the mask, enhancing image realism. Additionally, if an empty mask is provided, our model automatically places the object in diverse natural locations and scales, accelerating the compositing workflow. Our model outperforms existing object placement and compositing models in various quality metrics and user studies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_04559 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Thinking Outside the BBox: Unconstrained Generative Object Compositing Tarrés, Gemma Canet Lin, Zhe Zhang, Zhifei Zhang, Jianming Song, Yizhi Ruta, Dan Gilbert, Andrew Collomosse, John Kim, Soo Ye Computer Vision and Pattern Recognition Artificial Intelligence Compositing an object into an image involves multiple non-trivial sub-tasks such as object placement and scaling, color/lighting harmonization, viewpoint/geometry adjustment, and shadow/reflection generation. Recent generative image compositing methods leverage diffusion models to handle multiple sub-tasks at once. However, existing models face limitations due to their reliance on masking the original object during training, which constrains their generation to the input mask. Furthermore, obtaining an accurate input mask specifying the location and scale of the object in a new image can be highly challenging. To overcome such limitations, we define a novel problem of unconstrained generative object compositing, i.e., the generation is not bounded by the mask, and train a diffusion-based model on a synthesized paired dataset. Our first-of-its-kind model is able to generate object effects such as shadows and reflections that go beyond the mask, enhancing image realism. Additionally, if an empty mask is provided, our model automatically places the object in diverse natural locations and scales, accelerating the compositing workflow. Our model outperforms existing object placement and compositing models in various quality metrics and user studies. |
| title | Thinking Outside the BBox: Unconstrained Generative Object Compositing |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2409.04559 |