Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.11119 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915969978335232 |
|---|---|
| author | Zhang, Tiantian Zuo, Jierui Chen, Michael Wang, Wenping |
| author_facet | Zhang, Tiantian Zuo, Jierui Chen, Michael Wang, Wenping |
| contents | Recent theory suggests that reward-model-first methods can be more sample-efficient than direct policy fitting when the reward function is statistically simpler than the induced policy. We propose DDO-RM, a finite-candidate decision-optimization method that converts reward scores into an explicit target distribution. Unlike PPO-based RLHF or DPO, DDO-RM performs a KL-regularized mirror-descent update to project the policy toward a reward-improved distribution over a candidate set. Preliminary experiments on Pythia-410M show that DDO-RM outperforms DPO in pair accuracy (0.52 to 0.56) and mean margin (0.13 to 0.53). Our framework provides a principled connection between reward learning and mirror-descent policy improvement. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_11119 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DDO-RM: Distribution-Level Policy Improvement after Reward Learning Zhang, Tiantian Zuo, Jierui Chen, Michael Wang, Wenping Machine Learning Recent theory suggests that reward-model-first methods can be more sample-efficient than direct policy fitting when the reward function is statistically simpler than the induced policy. We propose DDO-RM, a finite-candidate decision-optimization method that converts reward scores into an explicit target distribution. Unlike PPO-based RLHF or DPO, DDO-RM performs a KL-regularized mirror-descent update to project the policy toward a reward-improved distribution over a candidate set. Preliminary experiments on Pythia-410M show that DDO-RM outperforms DPO in pair accuracy (0.52 to 0.56) and mean margin (0.13 to 0.53). Our framework provides a principled connection between reward learning and mirror-descent policy improvement. |
| title | DDO-RM: Distribution-Level Policy Improvement after Reward Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2604.11119 |