PGPO: Enhancing Agent Reasoning via Pseudocode-style Planning Guided Preference Optimization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915318181396480 |
|---|---|
| author | Cao, Zouying Wang, Runze Yang, Yifei Ma, Xinbei Zhu, Xiaoyong Zheng, Bo Zhao, Hai |
| author_facet | Cao, Zouying Wang, Runze Yang, Yifei Ma, Xinbei Zhu, Xiaoyong Zheng, Bo Zhao, Hai |
| contents | Large Language Model (LLM) agents have demonstrated impressive capabilities in handling complex interactive problems. Existing LLM agents mainly generate natural language plans to guide reasoning, which is verbose and inefficient. NL plans are also tailored to specific tasks and restrict agents' ability to generalize across similar tasks. To this end, we explore pseudocode-style plans (P-code Plan) to capture the structural logic of reasoning. We find that P-code Plan empowers LLM agents with stronger generalization ability and more efficiency. Inspired by this finding, we propose a pseudocode-style Planning Guided Preference Optimization method called PGPO for effective agent learning. With two planning-oriented rewards, PGPO further enhances LLM agents' ability to generate high-quality P-code Plans and subsequent reasoning. Experiments show that PGPO achieves superior performance on representative agent benchmarks and outperforms the current leading baselines. Analyses reveal the advantage of PGPO in reducing action errors and omissions during reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_01475 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PGPO: Enhancing Agent Reasoning via Pseudocode-style Planning Guided Preference Optimization Cao, Zouying Wang, Runze Yang, Yifei Ma, Xinbei Zhu, Xiaoyong Zheng, Bo Zhao, Hai Artificial Intelligence Computation and Language Large Language Model (LLM) agents have demonstrated impressive capabilities in handling complex interactive problems. Existing LLM agents mainly generate natural language plans to guide reasoning, which is verbose and inefficient. NL plans are also tailored to specific tasks and restrict agents' ability to generalize across similar tasks. To this end, we explore pseudocode-style plans (P-code Plan) to capture the structural logic of reasoning. We find that P-code Plan empowers LLM agents with stronger generalization ability and more efficiency. Inspired by this finding, we propose a pseudocode-style Planning Guided Preference Optimization method called PGPO for effective agent learning. With two planning-oriented rewards, PGPO further enhances LLM agents' ability to generate high-quality P-code Plans and subsequent reasoning. Experiments show that PGPO achieves superior performance on representative agent benchmarks and outperforms the current leading baselines. Analyses reveal the advantage of PGPO in reducing action errors and omissions during reasoning. |
| title | PGPO: Enhancing Agent Reasoning via Pseudocode-style Planning Guided Preference Optimization |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2506.01475 |