Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.14868 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912240915972096 |
|---|---|
| author | Shi, Wentao Yuan, Mengqi Wu, Junkang Wang, Qifan Feng, Fuli |
| author_facet | Shi, Wentao Yuan, Mengqi Wu, Junkang Wang, Qifan Feng, Fuli |
| contents | Adapting Large Language Models (LLMs) for agent tasks is critical in developing language agents. Direct Preference Optimization (DPO) is a promising technique for this adaptation with the alleviation of compounding errors, offering a means to directly optimize Reinforcement Learning (RL) objectives. However, applying DPO to multi-turn tasks presents challenges due to the inability to cancel the partition function. Overcoming this obstacle involves making the partition function independent of the current state and addressing length disparities between preferred and dis-preferred trajectories. In this light, we replace the policy constraint with the state-action occupancy measure constraint in the RL objective and add length normalization to the Bradley-Terry model, yielding a novel loss function named DMPO for multi-turn agent tasks with theoretical explanations. Extensive experiments on three multi-turn agent task datasets confirm the effectiveness and superiority of the DMPO loss. The code is available at https://github.com/swt-user/DMPO. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14868 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Direct Multi-Turn Preference Optimization for Language Agents Shi, Wentao Yuan, Mengqi Wu, Junkang Wang, Qifan Feng, Fuli Computation and Language Machine Learning Adapting Large Language Models (LLMs) for agent tasks is critical in developing language agents. Direct Preference Optimization (DPO) is a promising technique for this adaptation with the alleviation of compounding errors, offering a means to directly optimize Reinforcement Learning (RL) objectives. However, applying DPO to multi-turn tasks presents challenges due to the inability to cancel the partition function. Overcoming this obstacle involves making the partition function independent of the current state and addressing length disparities between preferred and dis-preferred trajectories. In this light, we replace the policy constraint with the state-action occupancy measure constraint in the RL objective and add length normalization to the Bradley-Terry model, yielding a novel loss function named DMPO for multi-turn agent tasks with theoretical explanations. Extensive experiments on three multi-turn agent task datasets confirm the effectiveness and superiority of the DMPO loss. The code is available at https://github.com/swt-user/DMPO. |
| title | Direct Multi-Turn Preference Optimization for Language Agents |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2406.14868 |