Feedback Efficient Online Fine-Tuning of Diffusion Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911960052793344 |
|---|---|
| author | Uehara, Masatoshi Zhao, Yulai Black, Kevin Hajiramezanali, Ehsan Scalia, Gabriele Diamant, Nathaniel Lee Tseng, Alex M Levine, Sergey Biancalani, Tommaso |
| author_facet | Uehara, Masatoshi Zhao, Yulai Black, Kevin Hajiramezanali, Ehsan Scalia, Gabriele Diamant, Nathaniel Lee Tseng, Alex M Levine, Sergey Biancalani, Tommaso |
| contents | Diffusion models excel at modeling complex data distributions, including those of images, proteins, and small molecules. However, in many cases, our goal is to model parts of the distribution that maximize certain properties: for example, we may want to generate images with high aesthetic quality, or molecules with high bioactivity. It is natural to frame this as a reinforcement learning (RL) problem, in which the objective is to fine-tune a diffusion model to maximize a reward function that corresponds to some property. Even with access to online queries of the ground-truth reward function, efficiently discovering high-reward samples can be challenging: they might have a low probability in the initial distribution, and there might be many infeasible samples that do not even have a well-defined reward (e.g., unnatural images or physically impossible molecules). In this work, we propose a novel reinforcement learning procedure that efficiently explores on the manifold of feasible samples. We present a theoretical analysis providing a regret guarantee, as well as empirical validation across three domains: images, biological sequences, and molecules. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_16359 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Feedback Efficient Online Fine-Tuning of Diffusion Models Uehara, Masatoshi Zhao, Yulai Black, Kevin Hajiramezanali, Ehsan Scalia, Gabriele Diamant, Nathaniel Lee Tseng, Alex M Levine, Sergey Biancalani, Tommaso Machine Learning Artificial Intelligence Quantitative Methods Diffusion models excel at modeling complex data distributions, including those of images, proteins, and small molecules. However, in many cases, our goal is to model parts of the distribution that maximize certain properties: for example, we may want to generate images with high aesthetic quality, or molecules with high bioactivity. It is natural to frame this as a reinforcement learning (RL) problem, in which the objective is to fine-tune a diffusion model to maximize a reward function that corresponds to some property. Even with access to online queries of the ground-truth reward function, efficiently discovering high-reward samples can be challenging: they might have a low probability in the initial distribution, and there might be many infeasible samples that do not even have a well-defined reward (e.g., unnatural images or physically impossible molecules). In this work, we propose a novel reinforcement learning procedure that efficiently explores on the manifold of feasible samples. We present a theoretical analysis providing a regret guarantee, as well as empirical validation across three domains: images, biological sequences, and molecules. |
| title | Feedback Efficient Online Fine-Tuning of Diffusion Models |
| topic | Machine Learning Artificial Intelligence Quantitative Methods |
| url | https://arxiv.org/abs/2402.16359 |