Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.05116 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909535973670912 |
|---|---|
| author | Hiranaka, Ayano Chen, Shang-Fu Lai, Chieh-Hsin Kim, Dongjun Murata, Naoki Shibuya, Takashi Liao, Wei-Hsiang Sun, Shao-Hua Mitsufuji, Yuki |
| author_facet | Hiranaka, Ayano Chen, Shang-Fu Lai, Chieh-Hsin Kim, Dongjun Murata, Naoki Shibuya, Takashi Liao, Wei-Hsiang Sun, Shao-Hua Mitsufuji, Yuki |
| contents | Controllable generation through Stable Diffusion (SD) fine-tuning aims to improve fidelity, safety, and alignment with human guidance. Existing reinforcement learning from human feedback methods usually rely on predefined heuristic reward functions or pretrained reward models built on large-scale datasets, limiting their applicability to scenarios where collecting such data is costly or difficult. To effectively and efficiently utilize human feedback, we develop a framework, HERO, which leverages online human feedback collected on the fly during model learning. Specifically, HERO features two key mechanisms: (1) Feedback-Aligned Representation Learning, an online training method that captures human feedback and provides informative learning signals for fine-tuning, and (2) Feedback-Guided Image Generation, which involves generating images from SD's refined initialization samples, enabling faster convergence towards the evaluator's intent. We demonstrate that HERO is 4x more efficient in online feedback for body part anomaly correction compared to the best existing method. Additionally, experiments show that HERO can effectively handle tasks like reasoning, counting, personalization, and reducing NSFW content with only 0.5K online feedback. The code and project page are available at https://hero-dm.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_05116 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HERO: Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuning Hiranaka, Ayano Chen, Shang-Fu Lai, Chieh-Hsin Kim, Dongjun Murata, Naoki Shibuya, Takashi Liao, Wei-Hsiang Sun, Shao-Hua Mitsufuji, Yuki Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Human-Computer Interaction Controllable generation through Stable Diffusion (SD) fine-tuning aims to improve fidelity, safety, and alignment with human guidance. Existing reinforcement learning from human feedback methods usually rely on predefined heuristic reward functions or pretrained reward models built on large-scale datasets, limiting their applicability to scenarios where collecting such data is costly or difficult. To effectively and efficiently utilize human feedback, we develop a framework, HERO, which leverages online human feedback collected on the fly during model learning. Specifically, HERO features two key mechanisms: (1) Feedback-Aligned Representation Learning, an online training method that captures human feedback and provides informative learning signals for fine-tuning, and (2) Feedback-Guided Image Generation, which involves generating images from SD's refined initialization samples, enabling faster convergence towards the evaluator's intent. We demonstrate that HERO is 4x more efficient in online feedback for body part anomaly correction compared to the best existing method. Additionally, experiments show that HERO can effectively handle tasks like reasoning, counting, personalization, and reducing NSFW content with only 0.5K online feedback. The code and project page are available at https://hero-dm.github.io/. |
| title | HERO: Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuning |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Human-Computer Interaction |
| url | https://arxiv.org/abs/2410.05116 |