On a Connection Between Imitation Learning and RLHF
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910862598471680 |
|---|---|
| author | Xiao, Teng Yuan, Yige Li, Mingxiao Chen, Zhengyu Honavar, Vasant G |
| author_facet | Xiao, Teng Yuan, Yige Li, Mingxiao Chen, Zhengyu Honavar, Vasant G |
| contents | This work studies the alignment of large language models with preference data from an imitation learning perspective. We establish a close theoretical connection between reinforcement learning from human feedback RLHF and imitation learning (IL), revealing that RLHF implicitly performs imitation learning on the preference data distribution. Building on this connection, we propose DIL, a principled framework that directly optimizes the imitation learning objective. DIL provides a unified imitation learning perspective on alignment, encompassing existing alignment algorithms as special cases while naturally introducing new variants. By bridging IL and RLHF, DIL offers new insights into alignment with RLHF. Extensive experiments demonstrate that DIL outperforms existing methods on various challenging benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_05079 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On a Connection Between Imitation Learning and RLHF Xiao, Teng Yuan, Yige Li, Mingxiao Chen, Zhengyu Honavar, Vasant G Machine Learning This work studies the alignment of large language models with preference data from an imitation learning perspective. We establish a close theoretical connection between reinforcement learning from human feedback RLHF and imitation learning (IL), revealing that RLHF implicitly performs imitation learning on the preference data distribution. Building on this connection, we propose DIL, a principled framework that directly optimizes the imitation learning objective. DIL provides a unified imitation learning perspective on alignment, encompassing existing alignment algorithms as special cases while naturally introducing new variants. By bridging IL and RLHF, DIL offers new insights into alignment with RLHF. Extensive experiments demonstrate that DIL outperforms existing methods on various challenging benchmarks. |
| title | On a Connection Between Imitation Learning and RLHF |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2503.05079 |