Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916453316296704 |
|---|---|
| author | Li, Xiner Zhao, Yulai Wang, Chenyu Scalia, Gabriele Eraslan, Gokcen Nair, Surag Biancalani, Tommaso Ji, Shuiwang Regev, Aviv Levine, Sergey Uehara, Masatoshi |
| author_facet | Li, Xiner Zhao, Yulai Wang, Chenyu Scalia, Gabriele Eraslan, Gokcen Nair, Surag Biancalani, Tommaso Ji, Shuiwang Regev, Aviv Levine, Sergey Uehara, Masatoshi |
| contents | Diffusion models excel at capturing the natural design spaces of images, molecules, DNA, RNA, and protein sequences. However, rather than merely generating designs that are natural, we often aim to optimize downstream reward functions while preserving the naturalness of these design spaces. Existing methods for achieving this goal often require ``differentiable'' proxy models (\textit{e.g.}, classifier guidance or DPS) or involve computationally expensive fine-tuning of diffusion models (\textit{e.g.}, classifier-free guidance, RL-based fine-tuning). In our work, we propose a new method to address these challenges. Our algorithm is an iterative sampling method that integrates soft value functions, which looks ahead to how intermediate noisy states lead to high rewards in the future, into the standard inference procedure of pre-trained diffusion models. Notably, our approach avoids fine-tuning generative models and eliminates the need to construct differentiable models. This enables us to (1) directly utilize non-differentiable features/reward feedback, commonly used in many scientific domains, and (2) apply our method to recent discrete diffusion models in a principled way. Finally, we demonstrate the effectiveness of our algorithm across several domains, including image generation, molecule generation, and DNA/RNA sequence generation. The code is available at \href{https://github.com/masa-ue/SVDD}{https://github.com/masa-ue/SVDD}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_08252 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding Li, Xiner Zhao, Yulai Wang, Chenyu Scalia, Gabriele Eraslan, Gokcen Nair, Surag Biancalani, Tommaso Ji, Shuiwang Regev, Aviv Levine, Sergey Uehara, Masatoshi Machine Learning Artificial Intelligence Genomics Diffusion models excel at capturing the natural design spaces of images, molecules, DNA, RNA, and protein sequences. However, rather than merely generating designs that are natural, we often aim to optimize downstream reward functions while preserving the naturalness of these design spaces. Existing methods for achieving this goal often require ``differentiable'' proxy models (\textit{e.g.}, classifier guidance or DPS) or involve computationally expensive fine-tuning of diffusion models (\textit{e.g.}, classifier-free guidance, RL-based fine-tuning). In our work, we propose a new method to address these challenges. Our algorithm is an iterative sampling method that integrates soft value functions, which looks ahead to how intermediate noisy states lead to high rewards in the future, into the standard inference procedure of pre-trained diffusion models. Notably, our approach avoids fine-tuning generative models and eliminates the need to construct differentiable models. This enables us to (1) directly utilize non-differentiable features/reward feedback, commonly used in many scientific domains, and (2) apply our method to recent discrete diffusion models in a principled way. Finally, we demonstrate the effectiveness of our algorithm across several domains, including image generation, molecule generation, and DNA/RNA sequence generation. The code is available at \href{https://github.com/masa-ue/SVDD}{https://github.com/masa-ue/SVDD}. |
| title | Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding |
| topic | Machine Learning Artificial Intelligence Genomics |
| url | https://arxiv.org/abs/2408.08252 |