AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914665588588544 |
|---|---|
| author | Dong, Zibin Yuan, Yifu Hao, Jianye Ni, Fei Mu, Yao Zheng, Yan Hu, Yujing Lv, Tangjie Fan, Changjie Hu, Zhipeng |
| author_facet | Dong, Zibin Yuan, Yifu Hao, Jianye Ni, Fei Mu, Yao Zheng, Yan Hu, Yujing Lv, Tangjie Fan, Changjie Hu, Zhipeng |
| contents | Aligning agent behaviors with diverse human preferences remains a challenging problem in reinforcement learning (RL), owing to the inherent abstractness and mutability of human preferences. To address these issues, we propose AlignDiff, a novel framework that leverages RL from Human Feedback (RLHF) to quantify human preferences, covering abstractness, and utilizes them to guide diffusion planning for zero-shot behavior customizing, covering mutability. AlignDiff can accurately match user-customized behaviors and efficiently switch from one to another. To build the framework, we first establish the multi-perspective human feedback datasets, which contain comparisons for the attributes of diverse behaviors, and then train an attribute strength model to predict quantified relative strengths. After relabeling behavioral datasets with relative strengths, we proceed to train an attribute-conditioned diffusion model, which serves as a planner with the attribute strength model as a director for preference aligning at the inference phase. We evaluate AlignDiff on various locomotion tasks and demonstrate its superior performance on preference matching, switching, and covering compared to other baselines. Its capability of completing unseen downstream tasks under human instructions also showcases the promising potential for human-AI collaboration. More visualization videos are released on https://aligndiff.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_02054 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model Dong, Zibin Yuan, Yifu Hao, Jianye Ni, Fei Mu, Yao Zheng, Yan Hu, Yujing Lv, Tangjie Fan, Changjie Hu, Zhipeng Artificial Intelligence Aligning agent behaviors with diverse human preferences remains a challenging problem in reinforcement learning (RL), owing to the inherent abstractness and mutability of human preferences. To address these issues, we propose AlignDiff, a novel framework that leverages RL from Human Feedback (RLHF) to quantify human preferences, covering abstractness, and utilizes them to guide diffusion planning for zero-shot behavior customizing, covering mutability. AlignDiff can accurately match user-customized behaviors and efficiently switch from one to another. To build the framework, we first establish the multi-perspective human feedback datasets, which contain comparisons for the attributes of diverse behaviors, and then train an attribute strength model to predict quantified relative strengths. After relabeling behavioral datasets with relative strengths, we proceed to train an attribute-conditioned diffusion model, which serves as a planner with the attribute strength model as a director for preference aligning at the inference phase. We evaluate AlignDiff on various locomotion tasks and demonstrate its superior performance on preference matching, switching, and covering compared to other baselines. Its capability of completing unseen downstream tasks under human instructions also showcases the promising potential for human-AI collaboration. More visualization videos are released on https://aligndiff.github.io/. |
| title | AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2310.02054 |