Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2312.03584 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913954023866368 |
|---|---|
| author | Najdenkoska, Ivona Sinha, Animesh Dubey, Abhimanyu Mahajan, Dhruv Ramanathan, Vignesh Radenovic, Filip |
| author_facet | Najdenkoska, Ivona Sinha, Animesh Dubey, Abhimanyu Mahajan, Dhruv Ramanathan, Vignesh Radenovic, Filip |
| contents | We propose Context Diffusion, a diffusion-based framework that enables image generation models to learn from visual examples presented in context. Recent work tackles such in-context learning for image generation, where a query image is provided alongside context examples and text prompts. However, the quality and context fidelity of the generated images deteriorate when the prompt is not present, demonstrating that these models cannot truly learn from the visual context. To address this, we propose a novel framework that separates the encoding of the visual context and the preservation of the desired image layout. This results in the ability to learn from the visual context and prompts, but also from either of them. Furthermore, we enable our model to handle few-shot settings, to effectively address diverse in-context learning scenarios. Our experiments and human evaluation demonstrate that Context Diffusion excels in both in-domain and out-of-domain tasks, resulting in an overall enhancement in image quality and context fidelity compared to counterpart models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_03584 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Context Diffusion: In-Context Aware Image Generation Najdenkoska, Ivona Sinha, Animesh Dubey, Abhimanyu Mahajan, Dhruv Ramanathan, Vignesh Radenovic, Filip Computer Vision and Pattern Recognition We propose Context Diffusion, a diffusion-based framework that enables image generation models to learn from visual examples presented in context. Recent work tackles such in-context learning for image generation, where a query image is provided alongside context examples and text prompts. However, the quality and context fidelity of the generated images deteriorate when the prompt is not present, demonstrating that these models cannot truly learn from the visual context. To address this, we propose a novel framework that separates the encoding of the visual context and the preservation of the desired image layout. This results in the ability to learn from the visual context and prompts, but also from either of them. Furthermore, we enable our model to handle few-shot settings, to effectively address diverse in-context learning scenarios. Our experiments and human evaluation demonstrate that Context Diffusion excels in both in-domain and out-of-domain tasks, resulting in an overall enhancement in image quality and context fidelity compared to counterpart models. |
| title | Context Diffusion: In-Context Aware Image Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.03584 |