IPAdapter-Instruct: Resolving Ambiguity in Image-based Conditioning using Instruct Prompts
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916370528075776 |
|---|---|
| author | Rowles, Ciara Vainer, Shimon De Nigris, Dante Elizarov, Slava Kutsy, Konstantin Donné, Simon |
| author_facet | Rowles, Ciara Vainer, Shimon De Nigris, Dante Elizarov, Slava Kutsy, Konstantin Donné, Simon |
| contents | Diffusion models continuously push the boundary of state-of-the-art image generation, but the process is hard to control with any nuance: practice proves that textual prompts are inadequate for accurately describing image style or fine structural details (such as faces). ControlNet and IPAdapter address this shortcoming by conditioning the generative process on imagery instead, but each individual instance is limited to modeling a single conditional posterior: for practical use-cases, where multiple different posteriors are desired within the same workflow, training and using multiple adapters is cumbersome. We propose IPAdapter-Instruct, which combines natural-image conditioning with ``Instruct'' prompts to swap between interpretations for the same conditioning image: style transfer, object extraction, both, or something else still? IPAdapterInstruct efficiently learns multiple tasks with minimal loss in quality compared to dedicated per-task models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_03209 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | IPAdapter-Instruct: Resolving Ambiguity in Image-based Conditioning using Instruct Prompts Rowles, Ciara Vainer, Shimon De Nigris, Dante Elizarov, Slava Kutsy, Konstantin Donné, Simon Computer Vision and Pattern Recognition Diffusion models continuously push the boundary of state-of-the-art image generation, but the process is hard to control with any nuance: practice proves that textual prompts are inadequate for accurately describing image style or fine structural details (such as faces). ControlNet and IPAdapter address this shortcoming by conditioning the generative process on imagery instead, but each individual instance is limited to modeling a single conditional posterior: for practical use-cases, where multiple different posteriors are desired within the same workflow, training and using multiple adapters is cumbersome. We propose IPAdapter-Instruct, which combines natural-image conditioning with ``Instruct'' prompts to swap between interpretations for the same conditioning image: style transfer, object extraction, both, or something else still? IPAdapterInstruct efficiently learns multiple tasks with minimal loss in quality compared to dedicated per-task models. |
| title | IPAdapter-Instruct: Resolving Ambiguity in Image-based Conditioning using Instruct Prompts |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.03209 |