Keypoint Abstraction using Large Models for Object-Relative Imitation Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912096254427136 |
|---|---|
| author | Fang, Xiaolin Huang, Bo-Ruei Mao, Jiayuan Shone, Jasmine Tenenbaum, Joshua B. Lozano-Pérez, Tomás Kaelbling, Leslie Pack |
| author_facet | Fang, Xiaolin Huang, Bo-Ruei Mao, Jiayuan Shone, Jasmine Tenenbaum, Joshua B. Lozano-Pérez, Tomás Kaelbling, Leslie Pack |
| contents | Generalization to novel object configurations and instances across diverse tasks and environments is a critical challenge in robotics. Keypoint-based representations have been proven effective as a succinct representation for capturing essential object features, and for establishing a reference frame in action prediction, enabling data-efficient learning of robot skills. However, their manual design nature and reliance on additional human labels limit their scalability. In this paper, we propose KALM, a framework that leverages large pre-trained vision-language models (LMs) to automatically generate task-relevant and cross-instance consistent keypoints. KALM distills robust and consistent keypoints across views and objects by generating proposals using LMs and verifies them against a small set of robot demonstration data. Based on the generated keypoints, we can train keypoint-conditioned policy models that predict actions in keypoint-centric frames, enabling robots to generalize effectively across varying object poses, camera views, and object instances with similar functional shapes. Our method demonstrates strong performance in the real world, adapting to different tasks and environments from only a handful of demonstrations while requiring no additional labels. Website: https://kalm-il.github.io/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_23254 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Keypoint Abstraction using Large Models for Object-Relative Imitation Learning Fang, Xiaolin Huang, Bo-Ruei Mao, Jiayuan Shone, Jasmine Tenenbaum, Joshua B. Lozano-Pérez, Tomás Kaelbling, Leslie Pack Robotics Artificial Intelligence Computer Vision and Pattern Recognition Generalization to novel object configurations and instances across diverse tasks and environments is a critical challenge in robotics. Keypoint-based representations have been proven effective as a succinct representation for capturing essential object features, and for establishing a reference frame in action prediction, enabling data-efficient learning of robot skills. However, their manual design nature and reliance on additional human labels limit their scalability. In this paper, we propose KALM, a framework that leverages large pre-trained vision-language models (LMs) to automatically generate task-relevant and cross-instance consistent keypoints. KALM distills robust and consistent keypoints across views and objects by generating proposals using LMs and verifies them against a small set of robot demonstration data. Based on the generated keypoints, we can train keypoint-conditioned policy models that predict actions in keypoint-centric frames, enabling robots to generalize effectively across varying object poses, camera views, and object instances with similar functional shapes. Our method demonstrates strong performance in the real world, adapting to different tasks and environments from only a handful of demonstrations while requiring no additional labels. Website: https://kalm-il.github.io/ |
| title | Keypoint Abstraction using Large Models for Object-Relative Imitation Learning |
| topic | Robotics Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.23254 |