Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917774947778560 |
|---|---|
| author | Zhao, Hanyang Chen, Haoxian Zhang, Ji Yao, David D. Tang, Wenpin |
| author_facet | Zhao, Hanyang Chen, Haoxian Zhang, Ji Yao, David D. Tang, Wenpin |
| contents | Reinforcement Learning from human feedback (RLHF) has been shown a promising direction for aligning generative models with human intent and has also been explored in recent works for alignment of diffusion generative models. In this work, we provide a rigorous treatment by formulating the task of fine-tuning diffusion models, with reward functions learned from human feedback, as an exploratory continuous-time stochastic control problem. Our key idea lies in treating the score-matching functions as controls/actions, and upon this, we develop a unified framework from a continuous-time perspective, to employ reinforcement learning (RL) algorithms in terms of improving the generation quality of diffusion models. We also develop the corresponding continuous-time RL theory for policy optimization and regularization under assumptions of stochastic different equations driven environment. Experiments on the text-to-image (T2I) generation will be reported in the accompanied paper. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_08400 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning Zhao, Hanyang Chen, Haoxian Zhang, Ji Yao, David D. Tang, Wenpin Machine Learning Artificial Intelligence Reinforcement Learning from human feedback (RLHF) has been shown a promising direction for aligning generative models with human intent and has also been explored in recent works for alignment of diffusion generative models. In this work, we provide a rigorous treatment by formulating the task of fine-tuning diffusion models, with reward functions learned from human feedback, as an exploratory continuous-time stochastic control problem. Our key idea lies in treating the score-matching functions as controls/actions, and upon this, we develop a unified framework from a continuous-time perspective, to employ reinforcement learning (RL) algorithms in terms of improving the generation quality of diffusion models. We also develop the corresponding continuous-time RL theory for policy optimization and regularization under assumptions of stochastic different equations driven environment. Experiments on the text-to-image (T2I) generation will be reported in the accompanied paper. |
| title | Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2409.08400 |