Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2502.12530 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911441541398528 |
|---|---|
| author | Yang, Xinyi Zeng, Liang Dong, Heng Yu, Chao Wu, Xiaoran Yang, Huazhong Wang, Yu Tambe, Milind Wang, Tonghan |
| author_facet | Yang, Xinyi Zeng, Liang Dong, Heng Yu, Chao Wu, Xiaoran Yang, Huazhong Wang, Yu Tambe, Milind Wang, Tonghan |
| contents | As humans increasingly share environments with diverse agents powered by RL, LLMs, and beyond, the ability to explain agent policies in natural language is vital for reliable coexistence. We introduce a general-purpose framework that trains explanation-generating LLMs via reinforcement learning from AI feedback, with distributional rewards generated by generative continuous normalizing flows (CNFs). CNFs capture the pluralistic and probabilistic nature of human judgments about explanations. Moreover, under mild assumptions, CNFs provably bound deviations from true human reward distributions when trained on noisy proxy rewards from LLMs. We design a specialized CNF architecture that selectively attends to linguistic cues in the decision context and explanations when generating rewards. Human and LLM evaluators find that our method delivers explanations that enable more accurate predictions of true agent decisions, exhibit greater logical soundness and actionability, and impose lower cognitive load than explanations trained with proxy LLM rewards or state-of-the-art RLHF and RLAIF baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_12530 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanations Yang, Xinyi Zeng, Liang Dong, Heng Yu, Chao Wu, Xiaoran Yang, Huazhong Wang, Yu Tambe, Milind Wang, Tonghan Computation and Language Machine Learning As humans increasingly share environments with diverse agents powered by RL, LLMs, and beyond, the ability to explain agent policies in natural language is vital for reliable coexistence. We introduce a general-purpose framework that trains explanation-generating LLMs via reinforcement learning from AI feedback, with distributional rewards generated by generative continuous normalizing flows (CNFs). CNFs capture the pluralistic and probabilistic nature of human judgments about explanations. Moreover, under mild assumptions, CNFs provably bound deviations from true human reward distributions when trained on noisy proxy rewards from LLMs. We design a specialized CNF architecture that selectively attends to linguistic cues in the decision context and explanations when generating rewards. Human and LLM evaluators find that our method delivers explanations that enable more accurate predictions of true agent decisions, exhibit greater logical soundness and actionability, and impose lower cognitive load than explanations trained with proxy LLM rewards or state-of-the-art RLHF and RLAIF baselines. |
| title | Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanations |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2502.12530 |