Inference-Time Policy Steering through Human Interactions
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917968233889792 |
|---|---|
| author | Wang, Yanwei Wang, Lirui Du, Yilun Sundaralingam, Balakumar Yang, Xuning Chao, Yu-Wei Perez-D'Arpino, Claudia Fox, Dieter Shah, Julie |
| author_facet | Wang, Yanwei Wang, Lirui Du, Yilun Sundaralingam, Balakumar Yang, Xuning Chao, Yu-Wei Perez-D'Arpino, Claudia Fox, Dieter Shah, Julie |
| contents | Generative policies trained with human demonstrations can autonomously accomplish multimodal, long-horizon tasks. However, during inference, humans are often removed from the policy execution loop, limiting the ability to guide a pre-trained policy towards a specific sub-goal or trajectory shape among multiple predictions. Naive human intervention may inadvertently exacerbate distribution shift, leading to constraint violations or execution failures. To better align policy output with human intent without inducing out-of-distribution errors, we propose an Inference-Time Policy Steering (ITPS) framework that leverages human interactions to bias the generative sampling process, rather than fine-tuning the policy on interaction data. We evaluate ITPS across three simulated and real-world benchmarks, testing three forms of human interaction and associated alignment distance metrics. Among six sampling strategies, our proposed stochastic sampling with diffusion policy achieves the best trade-off between alignment and distribution shift. Videos are available at https://yanweiw.github.io/itps/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_16627 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Inference-Time Policy Steering through Human Interactions Wang, Yanwei Wang, Lirui Du, Yilun Sundaralingam, Balakumar Yang, Xuning Chao, Yu-Wei Perez-D'Arpino, Claudia Fox, Dieter Shah, Julie Robotics Artificial Intelligence Human-Computer Interaction Machine Learning Generative policies trained with human demonstrations can autonomously accomplish multimodal, long-horizon tasks. However, during inference, humans are often removed from the policy execution loop, limiting the ability to guide a pre-trained policy towards a specific sub-goal or trajectory shape among multiple predictions. Naive human intervention may inadvertently exacerbate distribution shift, leading to constraint violations or execution failures. To better align policy output with human intent without inducing out-of-distribution errors, we propose an Inference-Time Policy Steering (ITPS) framework that leverages human interactions to bias the generative sampling process, rather than fine-tuning the policy on interaction data. We evaluate ITPS across three simulated and real-world benchmarks, testing three forms of human interaction and associated alignment distance metrics. Among six sampling strategies, our proposed stochastic sampling with diffusion policy achieves the best trade-off between alignment and distribution shift. Videos are available at https://yanweiw.github.io/itps/. |
| title | Inference-Time Policy Steering through Human Interactions |
| topic | Robotics Artificial Intelligence Human-Computer Interaction Machine Learning |
| url | https://arxiv.org/abs/2411.16627 |