RLTHF: Targeted Human Feedback for LLM Alignment
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909725951524864 |
|---|---|
| author | Xu, Yifei Chakraborty, Tusher Kıcıman, Emre Aryal, Bibek Rodrigues, Eduardo Sharma, Srinagesh Estevao, Roberto Balaguer, Maria Angels de Luis Wolk, Jessica Padilha, Rafael Nunes, Leonardo Balakrishnan, Shobana Lu, Songwu Chandra, Ranveer |
| author_facet | Xu, Yifei Chakraborty, Tusher Kıcıman, Emre Aryal, Bibek Rodrigues, Eduardo Sharma, Srinagesh Estevao, Roberto Balaguer, Maria Angels de Luis Wolk, Jessica Padilha, Rafael Nunes, Leonardo Balakrishnan, Shobana Lu, Songwu Chandra, Ranveer |
| contents | Fine-tuning large language models (LLMs) to align with user preferences is challenging due to the high cost of quality human annotations in Reinforcement Learning from Human Feedback (RLHF) and the generalizability limitations of AI Feedback. To address these challenges, we propose RLTHF, a human-AI hybrid framework that combines LLM-based initial alignment with selective human annotations to achieve full-human annotation alignment with minimal effort. RLTHF identifies hard-to-annotate samples mislabeled by LLMs using a reward model's reward distribution and iteratively enhances alignment by integrating strategic human corrections while leveraging LLM's correctly labeled samples. Evaluations on HH-RLHF and TL;DR datasets show that RLTHF reaches full-human annotation-level alignment with only 6-7% of the human annotation effort. Furthermore, models trained on RLTHF's curated datasets for downstream tasks outperform those trained on fully human-annotated datasets, underscoring the effectiveness of RLTHF. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13417 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RLTHF: Targeted Human Feedback for LLM Alignment Xu, Yifei Chakraborty, Tusher Kıcıman, Emre Aryal, Bibek Rodrigues, Eduardo Sharma, Srinagesh Estevao, Roberto Balaguer, Maria Angels de Luis Wolk, Jessica Padilha, Rafael Nunes, Leonardo Balakrishnan, Shobana Lu, Songwu Chandra, Ranveer Computation and Language Artificial Intelligence Machine Learning Fine-tuning large language models (LLMs) to align with user preferences is challenging due to the high cost of quality human annotations in Reinforcement Learning from Human Feedback (RLHF) and the generalizability limitations of AI Feedback. To address these challenges, we propose RLTHF, a human-AI hybrid framework that combines LLM-based initial alignment with selective human annotations to achieve full-human annotation alignment with minimal effort. RLTHF identifies hard-to-annotate samples mislabeled by LLMs using a reward model's reward distribution and iteratively enhances alignment by integrating strategic human corrections while leveraging LLM's correctly labeled samples. Evaluations on HH-RLHF and TL;DR datasets show that RLTHF reaches full-human annotation-level alignment with only 6-7% of the human annotation effort. Furthermore, models trained on RLTHF's curated datasets for downstream tasks outperform those trained on fully human-annotated datasets, underscoring the effectiveness of RLTHF. |
| title | RLTHF: Targeted Human Feedback for LLM Alignment |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2502.13417 |