Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916991145607168 |
|---|---|
| author | Joselowitz, Jared Majumdar, Ritam Jagota, Arjun Bou, Matthieu Patel, Nyal Krishna, Satyapriya Parbhoo, Sonali |
| author_facet | Joselowitz, Jared Majumdar, Ritam Jagota, Arjun Bou, Matthieu Patel, Nyal Krishna, Satyapriya Parbhoo, Sonali |
| contents | Large language models (LLMs) trained with Reinforcement Learning from Human Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying reward functions and decision-making processes remain opaque. This paper introduces a novel approach to interpreting LLMs by applying inverse reinforcement learning (IRL) to recover their implicit reward functions. We conduct experiments on toxicity-aligned LLMs of varying sizes, extracting reward models that achieve up to 85% accuracy in predicting human preferences. Our analysis reveals key insights into the non-identifiability of reward functions, the relationship between model size and interpretability, and potential pitfalls in the RLHF process. We demonstrate that IRL-derived reward models can be used to fine-tune new LLMs, resulting in comparable or improved performance on toxicity benchmarks. This work provides a new lens for understanding and improving LLM alignment, with implications for the responsible development and deployment of these powerful systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12491 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning Joselowitz, Jared Majumdar, Ritam Jagota, Arjun Bou, Matthieu Patel, Nyal Krishna, Satyapriya Parbhoo, Sonali Computation and Language Large language models (LLMs) trained with Reinforcement Learning from Human Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying reward functions and decision-making processes remain opaque. This paper introduces a novel approach to interpreting LLMs by applying inverse reinforcement learning (IRL) to recover their implicit reward functions. We conduct experiments on toxicity-aligned LLMs of varying sizes, extracting reward models that achieve up to 85% accuracy in predicting human preferences. Our analysis reveals key insights into the non-identifiability of reward functions, the relationship between model size and interpretability, and potential pitfalls in the RLHF process. We demonstrate that IRL-derived reward models can be used to fine-tune new LLMs, resulting in comparable or improved performance on toxicity benchmarks. This work provides a new lens for understanding and improving LLM alignment, with implications for the responsible development and deployment of these powerful systems. |
| title | Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.12491 |