Improving GFlowNets for Text-to-Image Diffusion Alignment
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910763314053120 |
|---|---|
| author | Zhang, Dinghuai Zhang, Yizhe Gu, Jiatao Zhang, Ruixiang Susskind, Josh Jaitly, Navdeep Zhai, Shuangfei |
| author_facet | Zhang, Dinghuai Zhang, Yizhe Gu, Jiatao Zhang, Ruixiang Susskind, Josh Jaitly, Navdeep Zhai, Shuangfei |
| contents | Diffusion models have become the de-facto approach for generating visual data, which are trained to match the distribution of the training dataset. In addition, we also want to control generation to fulfill desired properties such as alignment to a text description, which can be specified with a black-box reward function. Prior works fine-tune pretrained diffusion models to achieve this goal through reinforcement learning-based algorithms. Nonetheless, they suffer from issues including slow credit assignment as well as low quality in their generated samples. In this work, we explore techniques that do not directly maximize the reward but rather generate high-reward images with relatively high probability -- a natural scenario for the framework of generative flow networks (GFlowNets). To this end, we propose the Diffusion Alignment with GFlowNet (DAG) algorithm to post-train diffusion models with black-box property functions. Extensive experiments on Stable Diffusion and various reward specifications corroborate that our method could effectively align large-scale text-to-image diffusion models with given reward information. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_00633 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving GFlowNets for Text-to-Image Diffusion Alignment Zhang, Dinghuai Zhang, Yizhe Gu, Jiatao Zhang, Ruixiang Susskind, Josh Jaitly, Navdeep Zhai, Shuangfei Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Diffusion models have become the de-facto approach for generating visual data, which are trained to match the distribution of the training dataset. In addition, we also want to control generation to fulfill desired properties such as alignment to a text description, which can be specified with a black-box reward function. Prior works fine-tune pretrained diffusion models to achieve this goal through reinforcement learning-based algorithms. Nonetheless, they suffer from issues including slow credit assignment as well as low quality in their generated samples. In this work, we explore techniques that do not directly maximize the reward but rather generate high-reward images with relatively high probability -- a natural scenario for the framework of generative flow networks (GFlowNets). To this end, we propose the Diffusion Alignment with GFlowNet (DAG) algorithm to post-train diffusion models with black-box property functions. Extensive experiments on Stable Diffusion and various reward specifications corroborate that our method could effectively align large-scale text-to-image diffusion models with given reward information. |
| title | Improving GFlowNets for Text-to-Image Diffusion Alignment |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.00633 |