DEPO: Dual-Efficiency Preference Optimization for LLM Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915626817159168 |
|---|---|
| author | Chen, Sirui Zhao, Mengshi Xu, Lei Zhao, Yuying Zhu, Beier Zhang, Hanwang Zhao, Shengjie Lu, Chaochao |
| author_facet | Chen, Sirui Zhao, Mengshi Xu, Lei Zhao, Yuying Zhu, Beier Zhang, Hanwang Zhao, Shengjie Lu, Chaochao |
| contents | Recent advances in large language models (LLMs) have greatly improved their reasoning and decision-making abilities when deployed as agents. Richer reasoning, however, often comes at the cost of longer chain of thought (CoT), hampering interaction efficiency in real-world scenarios. Nevertheless, there still lacks systematic definition of LLM agent efficiency, hindering targeted improvements. To this end, we introduce dual-efficiency, comprising (i) step-level efficiency, which minimizes tokens per step, and (ii) trajectory-level efficiency, which minimizes the number of steps to complete a task. Building on this definition, we propose DEPO, a dual-efficiency preference optimization method that jointly rewards succinct responses and fewer action steps. Experiments on WebShop and BabyAI show that DEPO cuts token usage by up to 60.9% and steps by up to 26.9%, while achieving up to a 29.3% improvement in performance. DEPO also generalizes to three out-of-domain math benchmarks and retains its efficiency gains when trained on only 25% of the data. Our project page is at https://opencausalab.github.io/DEPO. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_15392 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DEPO: Dual-Efficiency Preference Optimization for LLM Agents Chen, Sirui Zhao, Mengshi Xu, Lei Zhao, Yuying Zhu, Beier Zhang, Hanwang Zhao, Shengjie Lu, Chaochao Computation and Language Artificial Intelligence Recent advances in large language models (LLMs) have greatly improved their reasoning and decision-making abilities when deployed as agents. Richer reasoning, however, often comes at the cost of longer chain of thought (CoT), hampering interaction efficiency in real-world scenarios. Nevertheless, there still lacks systematic definition of LLM agent efficiency, hindering targeted improvements. To this end, we introduce dual-efficiency, comprising (i) step-level efficiency, which minimizes tokens per step, and (ii) trajectory-level efficiency, which minimizes the number of steps to complete a task. Building on this definition, we propose DEPO, a dual-efficiency preference optimization method that jointly rewards succinct responses and fewer action steps. Experiments on WebShop and BabyAI show that DEPO cuts token usage by up to 60.9% and steps by up to 26.9%, while achieving up to a 29.3% improvement in performance. DEPO also generalizes to three out-of-domain math benchmarks and retains its efficiency gains when trained on only 25% of the data. Our project page is at https://opencausalab.github.io/DEPO. |
| title | DEPO: Dual-Efficiency Preference Optimization for LLM Agents |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2511.15392 |