Distribution Matching Distillation Meets Reinforcement Learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908911278227456 |
|---|---|
| author | Jiang, Dengyang Liu, Dongyang Wang, Zanyi Wu, Qilong Li, Liuzhuozheng Li, Hengzhuang Jin, Xin Liu, David Lu, Changsheng Li, Zhen Zhang, Bo Wang, Mengmeng Hoi, Steven Gao, Peng Yang, Harry |
| author_facet | Jiang, Dengyang Liu, Dongyang Wang, Zanyi Wu, Qilong Li, Liuzhuozheng Li, Hengzhuang Jin, Xin Liu, David Lu, Changsheng Li, Zhen Zhang, Bo Wang, Mengmeng Hoi, Steven Gao, Peng Yang, Harry |
| contents | Distribution Matching Distillation (DMD) facilitates efficient inference by distilling multi-step diffusion models into few-step variants. Concurrently, Reinforcement Learning (RL) has emerged as a vital tool for aligning generative models with human preferences. While both represent critical post-training stages for large-scale diffusion models, existing studies typically treat them as independent, sequential processes, leaving a systematic framework for their unification largely unexplored. In this work, we demonstrate that jointly optimizing these two objectives yields mutual benefits: RL enables more preference-aware and controllable distillation rather than uniformly compressing the full data distribution, while DMD serves as an effective regularizer to mitigate reward hacking during RL training. Building on these insights, we propose DMDR, a unified framework that incorporates Reward-Tilted Distribution Matching optimization alongside two dynamic distillation training strategies in the initial stage, followed by the joint DMD and RL optimization in the second stage. Extensive experiments demonstrate that DMDR achieves state-of-the-art visual quality and prompt adherence among few-step generation methods, even surpassing the performance of its multi-step teacher model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_13649 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Distribution Matching Distillation Meets Reinforcement Learning Jiang, Dengyang Liu, Dongyang Wang, Zanyi Wu, Qilong Li, Liuzhuozheng Li, Hengzhuang Jin, Xin Liu, David Lu, Changsheng Li, Zhen Zhang, Bo Wang, Mengmeng Hoi, Steven Gao, Peng Yang, Harry Computer Vision and Pattern Recognition Distribution Matching Distillation (DMD) facilitates efficient inference by distilling multi-step diffusion models into few-step variants. Concurrently, Reinforcement Learning (RL) has emerged as a vital tool for aligning generative models with human preferences. While both represent critical post-training stages for large-scale diffusion models, existing studies typically treat them as independent, sequential processes, leaving a systematic framework for their unification largely unexplored. In this work, we demonstrate that jointly optimizing these two objectives yields mutual benefits: RL enables more preference-aware and controllable distillation rather than uniformly compressing the full data distribution, while DMD serves as an effective regularizer to mitigate reward hacking during RL training. Building on these insights, we propose DMDR, a unified framework that incorporates Reward-Tilted Distribution Matching optimization alongside two dynamic distillation training strategies in the initial stage, followed by the joint DMD and RL optimization in the second stage. Extensive experiments demonstrate that DMDR achieves state-of-the-art visual quality and prompt adherence among few-step generation methods, even surpassing the performance of its multi-step teacher model. |
| title | Distribution Matching Distillation Meets Reinforcement Learning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.13649 |