Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913123633463296 |
|---|---|
| author | Bai, Yang Liu, Kaiyuan Zhuang, Ziyuan Zhou, Jiahong Weng, Rongxiang Chen, Xin Wang, Jingang Cai, Xunliang |
| author_facet | Bai, Yang Liu, Kaiyuan Zhuang, Ziyuan Zhou, Jiahong Weng, Rongxiang Chen, Xin Wang, Jingang Cai, Xunliang |
| contents | Complex reinforcement learning environments frequently employ multi-task and mixed-reward formulations. In these settings, heterogeneous reward distributions and correlated reward dimensions often destabilize the construction of scalar advantages. To address these challenges, we propose Reward-Decorrelated Policy Optimization (RDPO), a reward-processing method designed to explicitly target both failure modes. RDPO first utilizes Magnitude-Aware Quantile normalization to stabilize prompt-level advantage allocation across binary, fractional, and continuous rewards. It then applies Mahalanobis whitening within each active reward subspace to mitigate correlation redundancy prior to aggregation. When applied during the post-training of LongCat-Flash, RDPO enhances instruction following, writing quality, and robustness to hard prompts while remaining broadly competitive on reasoning and coding evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_13641 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization Bai, Yang Liu, Kaiyuan Zhuang, Ziyuan Zhou, Jiahong Weng, Rongxiang Chen, Xin Wang, Jingang Cai, Xunliang Machine Learning Computation and Language Complex reinforcement learning environments frequently employ multi-task and mixed-reward formulations. In these settings, heterogeneous reward distributions and correlated reward dimensions often destabilize the construction of scalar advantages. To address these challenges, we propose Reward-Decorrelated Policy Optimization (RDPO), a reward-processing method designed to explicitly target both failure modes. RDPO first utilizes Magnitude-Aware Quantile normalization to stabilize prompt-level advantage allocation across binary, fractional, and continuous rewards. It then applies Mahalanobis whitening within each active reward subspace to mitigate correlation redundancy prior to aggregation. When applied during the post-training of LongCat-Flash, RDPO enhances instruction following, writing quality, and robustness to hard prompts while remaining broadly competitive on reasoning and coding evaluations. |
| title | Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2605.13641 |