Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.02125 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915972685758464 |
|---|---|
| author | Zhao, Guangyu Lian, Kewei Ru, Haoxuan Zhang, Borong Lin, Haowei Mu, Zhancun Fu, Haobo Fu, Qiang Cai, Shaofei Wang, Zihao Liang, Yitao |
| author_facet | Zhao, Guangyu Lian, Kewei Ru, Haoxuan Zhang, Borong Lin, Haowei Mu, Zhancun Fu, Haobo Fu, Qiang Cai, Shaofei Wang, Zihao Liang, Yitao |
| contents | Goal-conditioned policies enable decision-making models to execute diverse behaviors based on specified goals, yet their downstream performance is often highly sensitive to the choice of instructions or prompts. To bypass the limitations of discrete text prompts, we formulate post-training adaptation as a latent control problem, where the goal embedding serves as a continuous control variable to modulate the behavior of a frozen policy. We propose Preference Goal Tuning (PGT), a framework that optimizes this latent control variable to align the induced trajectory distribution with task preferences. Unlike standard fine-tuning that updates policy parameters, PGT keeps the policy frozen and updates only the latent goal using a trajectory-level preference objective. This approach essentially searches for the optimal conditioning input that maximizes the likelihood of preferred behaviors while suppressing undesirable ones. We evaluate PGT on the Minecraft SkillForge benchmark across 17 tasks. With minimal data, PGT achieves average relative improvements of 72.0\% and 81.6\% on two foundation policies, consistently outperforming expert-crafted prompts. Crucially, by decoupling task alignment (latent goal) from physical dynamics (frozen policy), PGT surpasses full fine-tuning by 13.4\% in out-of-distribution settings, demonstrating superior robustness and generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_02125 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Preference Goal Tuning: Post-Training as Latent Control for Frozen Policies Zhao, Guangyu Lian, Kewei Ru, Haoxuan Zhang, Borong Lin, Haowei Mu, Zhancun Fu, Haobo Fu, Qiang Cai, Shaofei Wang, Zihao Liang, Yitao Artificial Intelligence Machine Learning Goal-conditioned policies enable decision-making models to execute diverse behaviors based on specified goals, yet their downstream performance is often highly sensitive to the choice of instructions or prompts. To bypass the limitations of discrete text prompts, we formulate post-training adaptation as a latent control problem, where the goal embedding serves as a continuous control variable to modulate the behavior of a frozen policy. We propose Preference Goal Tuning (PGT), a framework that optimizes this latent control variable to align the induced trajectory distribution with task preferences. Unlike standard fine-tuning that updates policy parameters, PGT keeps the policy frozen and updates only the latent goal using a trajectory-level preference objective. This approach essentially searches for the optimal conditioning input that maximizes the likelihood of preferred behaviors while suppressing undesirable ones. We evaluate PGT on the Minecraft SkillForge benchmark across 17 tasks. With minimal data, PGT achieves average relative improvements of 72.0\% and 81.6\% on two foundation policies, consistently outperforming expert-crafted prompts. Crucially, by decoupling task alignment (latent goal) from physical dynamics (frozen policy), PGT surpasses full fine-tuning by 13.4\% in out-of-distribution settings, demonstrating superior robustness and generalization. |
| title | Preference Goal Tuning: Post-Training as Latent Control for Frozen Policies |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2412.02125 |