Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.11450 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916267221319680 |
|---|---|
| author | Liang, Jacky Xia, Fei Yu, Wenhao Zeng, Andy Arenas, Montserrat Gonzalez Attarian, Maria Bauza, Maria Bennice, Matthew Bewley, Alex Dostmohamed, Adil Fu, Chuyuan Kelly Gileadi, Nimrod Giustina, Marissa Gopalakrishnan, Keerthana Hasenclever, Leonard Humplik, Jan Hsu, Jasmine Joshi, Nikhil Jyenis, Ben Kew, Chase Kirmani, Sean Lee, Tsang-Wei Edward Lee, Kuang-Huei Michaely, Assaf Hurwitz Moore, Joss Oslund, Ken Rao, Dushyant Ren, Allen Tabanpour, Baruch Vuong, Quan Wahid, Ayzaan Xiao, Ted Xu, Ying Zhuang, Vincent Xu, Peng Frey, Erik Caluwaerts, Ken Zhang, Tingnan Ichter, Brian Tompson, Jonathan Takayama, Leila Vanhoucke, Vincent Shafran, Izhak Mataric, Maja Sadigh, Dorsa Heess, Nicolas Rao, Kanishka Stewart, Nik Tan, Jie Parada, Carolina |
| author_facet | Liang, Jacky Xia, Fei Yu, Wenhao Zeng, Andy Arenas, Montserrat Gonzalez Attarian, Maria Bauza, Maria Bennice, Matthew Bewley, Alex Dostmohamed, Adil Fu, Chuyuan Kelly Gileadi, Nimrod Giustina, Marissa Gopalakrishnan, Keerthana Hasenclever, Leonard Humplik, Jan Hsu, Jasmine Joshi, Nikhil Jyenis, Ben Kew, Chase Kirmani, Sean Lee, Tsang-Wei Edward Lee, Kuang-Huei Michaely, Assaf Hurwitz Moore, Joss Oslund, Ken Rao, Dushyant Ren, Allen Tabanpour, Baruch Vuong, Quan Wahid, Ayzaan Xiao, Ted Xu, Ying Zhuang, Vincent Xu, Peng Frey, Erik Caluwaerts, Ken Zhang, Tingnan Ichter, Brian Tompson, Jonathan Takayama, Leila Vanhoucke, Vincent Shafran, Izhak Mataric, Maja Sadigh, Dorsa Heess, Nicolas Rao, Kanishka Stewart, Nik Tan, Jie Parada, Carolina |
| contents | Large language models (LLMs) have been shown to exhibit a wide range of capabilities, such as writing robot code from language commands -- enabling non-experts to direct robot behaviors, modify them based on feedback, or compose them to perform new tasks. However, these capabilities (driven by in-context learning) are limited to short-term interactions, where users' feedback remains relevant for only as long as it fits within the context size of the LLM, and can be forgotten over longer interactions. In this work, we investigate fine-tuning the robot code-writing LLMs, to remember their in-context interactions and improve their teachability i.e., how efficiently they adapt to human inputs (measured by average number of corrections before the user considers the task successful). Our key observation is that when human-robot interactions are viewed as a partially observable Markov decision process (in which human language inputs are observations, and robot code outputs are actions), then training an LLM to complete previous interactions is training a transition dynamics model -- that can be combined with classic robotics techniques such as model predictive control (MPC) to discover shorter paths to success. This gives rise to Language Model Predictive Control (LMPC), a framework that fine-tunes PaLM 2 to improve its teachability on 78 tasks across 5 robot embodiments -- improving non-expert teaching success rates of unseen tasks by 26.9% while reducing the average number of human corrections from 2.4 to 1.9. Experiments show that LMPC also produces strong meta-learners, improving the success rate of in-context learning new tasks on unseen robot embodiments and APIs by 31.5%. See videos, code, and demos at: https://robot-teaching.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_11450 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning to Learn Faster from Human Feedback with Language Model Predictive Control Liang, Jacky Xia, Fei Yu, Wenhao Zeng, Andy Arenas, Montserrat Gonzalez Attarian, Maria Bauza, Maria Bennice, Matthew Bewley, Alex Dostmohamed, Adil Fu, Chuyuan Kelly Gileadi, Nimrod Giustina, Marissa Gopalakrishnan, Keerthana Hasenclever, Leonard Humplik, Jan Hsu, Jasmine Joshi, Nikhil Jyenis, Ben Kew, Chase Kirmani, Sean Lee, Tsang-Wei Edward Lee, Kuang-Huei Michaely, Assaf Hurwitz Moore, Joss Oslund, Ken Rao, Dushyant Ren, Allen Tabanpour, Baruch Vuong, Quan Wahid, Ayzaan Xiao, Ted Xu, Ying Zhuang, Vincent Xu, Peng Frey, Erik Caluwaerts, Ken Zhang, Tingnan Ichter, Brian Tompson, Jonathan Takayama, Leila Vanhoucke, Vincent Shafran, Izhak Mataric, Maja Sadigh, Dorsa Heess, Nicolas Rao, Kanishka Stewart, Nik Tan, Jie Parada, Carolina Robotics Large language models (LLMs) have been shown to exhibit a wide range of capabilities, such as writing robot code from language commands -- enabling non-experts to direct robot behaviors, modify them based on feedback, or compose them to perform new tasks. However, these capabilities (driven by in-context learning) are limited to short-term interactions, where users' feedback remains relevant for only as long as it fits within the context size of the LLM, and can be forgotten over longer interactions. In this work, we investigate fine-tuning the robot code-writing LLMs, to remember their in-context interactions and improve their teachability i.e., how efficiently they adapt to human inputs (measured by average number of corrections before the user considers the task successful). Our key observation is that when human-robot interactions are viewed as a partially observable Markov decision process (in which human language inputs are observations, and robot code outputs are actions), then training an LLM to complete previous interactions is training a transition dynamics model -- that can be combined with classic robotics techniques such as model predictive control (MPC) to discover shorter paths to success. This gives rise to Language Model Predictive Control (LMPC), a framework that fine-tunes PaLM 2 to improve its teachability on 78 tasks across 5 robot embodiments -- improving non-expert teaching success rates of unseen tasks by 26.9% while reducing the average number of human corrections from 2.4 to 1.9. Experiments show that LMPC also produces strong meta-learners, improving the success rate of in-context learning new tasks on unseen robot embodiments and APIs by 31.5%. See videos, code, and demos at: https://robot-teaching.github.io/. |
| title | Learning to Learn Faster from Human Feedback with Language Model Predictive Control |
| topic | Robotics |
| url | https://arxiv.org/abs/2402.11450 |