Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.04465 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913542458834944 |
|---|---|
| author | Li, Shufan Kallidromitis, Konstantinos Gokul, Akash Kato, Yusuke Kozuka, Kazuki |
| author_facet | Li, Shufan Kallidromitis, Konstantinos Gokul, Akash Kato, Yusuke Kozuka, Kazuki |
| contents | We present Diffusion-KTO, a novel approach for aligning text-to-image diffusion models by formulating the alignment objective as the maximization of expected human utility. Since this objective applies to each generation independently, Diffusion-KTO does not require collecting costly pairwise preference data nor training a complex reward model. Instead, our objective requires simple per-image binary feedback signals, e.g. likes or dislikes, which are abundantly available. After fine-tuning using Diffusion-KTO, text-to-image diffusion models exhibit superior performance compared to existing techniques, including supervised fine-tuning and Diffusion-DPO, both in terms of human judgment and automatic evaluation metrics such as PickScore and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging readily available per-image binary signals and broadens the applicability of aligning text-to-image diffusion models with human preferences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_04465 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Aligning Diffusion Models by Optimizing Human Utility Li, Shufan Kallidromitis, Konstantinos Gokul, Akash Kato, Yusuke Kozuka, Kazuki Computer Vision and Pattern Recognition We present Diffusion-KTO, a novel approach for aligning text-to-image diffusion models by formulating the alignment objective as the maximization of expected human utility. Since this objective applies to each generation independently, Diffusion-KTO does not require collecting costly pairwise preference data nor training a complex reward model. Instead, our objective requires simple per-image binary feedback signals, e.g. likes or dislikes, which are abundantly available. After fine-tuning using Diffusion-KTO, text-to-image diffusion models exhibit superior performance compared to existing techniques, including supervised fine-tuning and Diffusion-DPO, both in terms of human judgment and automatic evaluation metrics such as PickScore and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging readily available per-image binary signals and broadens the applicability of aligning text-to-image diffusion models with human preferences. |
| title | Aligning Diffusion Models by Optimizing Human Utility |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.04465 |