Shape-Guided Diffusion with Inside-Outside Attention
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2022
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913293127385088 |
|---|---|
| author | Park, Dong Huk Luo, Grace Toste, Clayton Azadi, Samaneh Liu, Xihui Karalashvili, Maka Rohrbach, Anna Darrell, Trevor |
| author_facet | Park, Dong Huk Luo, Grace Toste, Clayton Azadi, Samaneh Liu, Xihui Karalashvili, Maka Rohrbach, Anna Darrell, Trevor |
| contents | We introduce precise object silhouette as a new form of user control in text-to-image diffusion models, which we dub Shape-Guided Diffusion. Our training-free method uses an Inside-Outside Attention mechanism during the inversion and generation process to apply a shape constraint to the cross- and self-attention maps. Our mechanism designates which spatial region is the object (inside) vs. background (outside) then associates edits to the correct region. We demonstrate the efficacy of our method on the shape-guided editing task, where the model must replace an object according to a text prompt and object mask. We curate a new ShapePrompts benchmark derived from MS-COCO and achieve SOTA results in shape faithfulness without a degradation in text alignment or image realism according to both automatic metrics and annotator ratings. Our data and code will be made available at https://shape-guided-diffusion.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2212_00210 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Shape-Guided Diffusion with Inside-Outside Attention Park, Dong Huk Luo, Grace Toste, Clayton Azadi, Samaneh Liu, Xihui Karalashvili, Maka Rohrbach, Anna Darrell, Trevor Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning We introduce precise object silhouette as a new form of user control in text-to-image diffusion models, which we dub Shape-Guided Diffusion. Our training-free method uses an Inside-Outside Attention mechanism during the inversion and generation process to apply a shape constraint to the cross- and self-attention maps. Our mechanism designates which spatial region is the object (inside) vs. background (outside) then associates edits to the correct region. We demonstrate the efficacy of our method on the shape-guided editing task, where the model must replace an object according to a text prompt and object mask. We curate a new ShapePrompts benchmark derived from MS-COCO and achieve SOTA results in shape faithfulness without a degradation in text alignment or image realism according to both automatic metrics and annotator ratings. Our data and code will be made available at https://shape-guided-diffusion.github.io. |
| title | Shape-Guided Diffusion with Inside-Outside Attention |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2212.00210 |