Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.20629 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908886857940992 |
|---|---|
| author | Chen, Chieh-Yun Wang, Zhonghao Chen, Qi Ye, Zhifan Shi, Min Zhao, Yue Zhao, Yinan Qu, Hui Lin, Wei-An Shen, Yiru Kale, Ajinkya Essa, Irfan Shi, Humphrey |
| author_facet | Chen, Chieh-Yun Wang, Zhonghao Chen, Qi Ye, Zhifan Shi, Min Zhao, Yue Zhao, Yinan Qu, Hui Lin, Wei-An Shen, Yiru Kale, Ajinkya Essa, Irfan Shi, Humphrey |
| contents | Reinforcement learning from human feedback (RLHF) with reward models has advanced alignment of generative models to human aesthetic and perceptual preferences. However, jointly optimizing multiple rewards often incurs an alignment tax, improving one dimension while degrading others. To address this, we introduce two complementary methods: MapReduce LoRA and Reward-aware Token Embedding (RaTE). MapReduce LoRA trains preference-specific LoRA experts in parallel and iteratively merges them to refine a shared base model; RaTE learns reward-specific token embeddings that compose at inference for flexible preference control. Experiments on Text-to-Image generation (Stable Diffusion 3.5 Medium and FLUX.1-dev) show improvements of 36.1%, 4.6%, and 55.7%, and 32.7%, 4.3%, and 67.1% on GenEval, PickScore, and OCR, respectively. On Text-to-Video generation (HunyuanVideo), visual and motion quality improve by 48.1% and 90.0%, respectively. On the language task, Helpful Assistant, with Llama-2 7B, helpful and harmless improve by 43.4% and 136.7%, respectively. Our framework sets a new state-of-the-art multi-preference alignment recipe across modalities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_20629 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models Chen, Chieh-Yun Wang, Zhonghao Chen, Qi Ye, Zhifan Shi, Min Zhao, Yue Zhao, Yinan Qu, Hui Lin, Wei-An Shen, Yiru Kale, Ajinkya Essa, Irfan Shi, Humphrey Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Reinforcement learning from human feedback (RLHF) with reward models has advanced alignment of generative models to human aesthetic and perceptual preferences. However, jointly optimizing multiple rewards often incurs an alignment tax, improving one dimension while degrading others. To address this, we introduce two complementary methods: MapReduce LoRA and Reward-aware Token Embedding (RaTE). MapReduce LoRA trains preference-specific LoRA experts in parallel and iteratively merges them to refine a shared base model; RaTE learns reward-specific token embeddings that compose at inference for flexible preference control. Experiments on Text-to-Image generation (Stable Diffusion 3.5 Medium and FLUX.1-dev) show improvements of 36.1%, 4.6%, and 55.7%, and 32.7%, 4.3%, and 67.1% on GenEval, PickScore, and OCR, respectively. On Text-to-Video generation (HunyuanVideo), visual and motion quality improve by 48.1% and 90.0%, respectively. On the language task, Helpful Assistant, with Llama-2 7B, helpful and harmless improve by 43.4% and 136.7%, respectively. Our framework sets a new state-of-the-art multi-preference alignment recipe across modalities. |
| title | MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2511.20629 |