DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916841463480320 |
|---|---|
| author | Jung, Sunghee Lee, Donghun Lee, Shinbok Seo, Gaeun Lee, Daniel Ko, Byeongil Cho, Junrae Kim, Kihyun Kim, Eunggyun Shin, Myeongcheol |
| author_facet | Jung, Sunghee Lee, Donghun Lee, Shinbok Seo, Gaeun Lee, Daniel Ko, Byeongil Cho, Junrae Kim, Kihyun Kim, Eunggyun Shin, Myeongcheol |
| contents | Tool-Augmented Larage Language Models (TA-LLMs) have shown promise in real-world applications, but face challenges in handling incomplete queries and out-of-scope requests. While existing approaches rely mainly on Supervised Fine-Tuning with expert trajectories, we propose DiaTool-DPO, a novel method that enhances TA-LLM's dialogue capabilities through Direct Preference Optimization. We model TA-LLM interactions as a Markov Decision Process with 5 distinct dialogue states and categorize user queries into 3 types based on their state transition trajectories. We automatically construct paired trajectory datasets of correct and incorrect dialogue flows and introduce a specialized objective loss for dialogue control. Our comprehensive evaluation demonstrates that DiaTool-DPO approaches GPT-4o's performance (94.8% in information gathering, 91% in tool call rejection) with substantial improvements over baseline (44% and 9.6% respectively) while maintaining core functionality. Our approach opens new possibilities for developing TA-LLMs that can handle diverse real-world scenarios without requiring additional expert demonstrations or human labeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_02882 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models Jung, Sunghee Lee, Donghun Lee, Shinbok Seo, Gaeun Lee, Daniel Ko, Byeongil Cho, Junrae Kim, Kihyun Kim, Eunggyun Shin, Myeongcheol Computation and Language Machine Learning Tool-Augmented Larage Language Models (TA-LLMs) have shown promise in real-world applications, but face challenges in handling incomplete queries and out-of-scope requests. While existing approaches rely mainly on Supervised Fine-Tuning with expert trajectories, we propose DiaTool-DPO, a novel method that enhances TA-LLM's dialogue capabilities through Direct Preference Optimization. We model TA-LLM interactions as a Markov Decision Process with 5 distinct dialogue states and categorize user queries into 3 types based on their state transition trajectories. We automatically construct paired trajectory datasets of correct and incorrect dialogue flows and introduce a specialized objective loss for dialogue control. Our comprehensive evaluation demonstrates that DiaTool-DPO approaches GPT-4o's performance (94.8% in information gathering, 91% in tool call rejection) with substantial improvements over baseline (44% and 9.6% respectively) while maintaining core functionality. Our approach opens new possibilities for developing TA-LLMs that can handle diverse real-world scenarios without requiring additional expert demonstrations or human labeling. |
| title | DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2504.02882 |