From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917551298052096 |
|---|---|
| author | Yang, Jun-Jie Hsu, Chia-Heng Chen, Kui-Yuan Hsieh, Ping-Chun |
| author_facet | Yang, Jun-Jie Hsu, Chia-Heng Chen, Kui-Yuan Hsieh, Ping-Chun |
| contents | Preference-based reinforcement learning (PbRL) avoids explicit reward engineering by learning from pairwise human preference feedback. Existing offline PbRL methods typically follow a two-stage pipeline, first learning a reward or preference model from labeled preferences and then performing offline RL on unlabeled data. We revisit offline PbRL through the lens of reward-free representation learning (RFRL) from the zero-shot RL literature, and propose a new training framework that first learns latent successor-measure representations from reward-free offline data, followed by contrastive search and fine-tuning using preference data. Through extensive experiments and ablations, we show that our method achieves superior preference efficiency over offline PbRL baselines. This work is the first to connect RFRL with PbRL, highlighting its potential as a feedback-efficient solution. Our code is publicly available at https://github.com/rl-bandits-lab/FB-PbRL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_01123 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning Yang, Jun-Jie Hsu, Chia-Heng Chen, Kui-Yuan Hsieh, Ping-Chun Machine Learning Preference-based reinforcement learning (PbRL) avoids explicit reward engineering by learning from pairwise human preference feedback. Existing offline PbRL methods typically follow a two-stage pipeline, first learning a reward or preference model from labeled preferences and then performing offline RL on unlabeled data. We revisit offline PbRL through the lens of reward-free representation learning (RFRL) from the zero-shot RL literature, and propose a new training framework that first learns latent successor-measure representations from reward-free offline data, followed by contrastive search and fine-tuning using preference data. Through extensive experiments and ablations, we show that our method achieves superior preference efficiency over offline PbRL baselines. This work is the first to connect RFRL with PbRL, highlighting its potential as a feedback-efficient solution. Our code is publicly available at https://github.com/rl-bandits-lab/FB-PbRL. |
| title | From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2606.01123 |