Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.06079 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912367943614464 |
|---|---|
| author | Huang, Shuaiyi Levy, Mara Gupta, Anubhav Ekpo, Daniel Zheng, Ruijie Shrivastava, Abhinav |
| author_facet | Huang, Shuaiyi Levy, Mara Gupta, Anubhav Ekpo, Daniel Zheng, Ruijie Shrivastava, Abhinav |
| contents | Preference feedback collected by human or VLM annotators is often noisy, presenting a significant challenge for preference-based reinforcement learning that relies on accurate preference labels. To address this challenge, we propose TREND, a novel framework that integrates few-shot expert demonstrations with a tri-teaching strategy for effective noise mitigation. Our method trains three reward models simultaneously, where each model views its small-loss preference pairs as useful knowledge and teaches such useful pairs to its peer network for updating the parameters. Remarkably, our approach requires as few as one to three expert demonstrations to achieve high performance. We evaluate TREND on various robotic manipulation tasks, achieving up to 90% success rates even with noise levels as high as 40%, highlighting its effective robustness in handling noisy preference feedback. Project page: https://shuaiyihuang.github.io/publications/TREND. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_06079 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrations Huang, Shuaiyi Levy, Mara Gupta, Anubhav Ekpo, Daniel Zheng, Ruijie Shrivastava, Abhinav Robotics Computer Vision and Pattern Recognition Preference feedback collected by human or VLM annotators is often noisy, presenting a significant challenge for preference-based reinforcement learning that relies on accurate preference labels. To address this challenge, we propose TREND, a novel framework that integrates few-shot expert demonstrations with a tri-teaching strategy for effective noise mitigation. Our method trains three reward models simultaneously, where each model views its small-loss preference pairs as useful knowledge and teaches such useful pairs to its peer network for updating the parameters. Remarkably, our approach requires as few as one to three expert demonstrations to achieve high performance. We evaluate TREND on various robotic manipulation tasks, achieving up to 90% success rates even with noise levels as high as 40%, highlighting its effective robustness in handling noisy preference feedback. Project page: https://shuaiyihuang.github.io/publications/TREND. |
| title | TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrations |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.06079 |