Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.12297 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916708558569472 |
|---|---|
| author | Khandate, Gagan Wang, Boxuan Park, Sarah Ni, Weizhe Palacios, Joaquin Lampo, Kathyrn Wu, Philippe Ho, Rosh Chang, Eric Ciocarlie, Matei |
| author_facet | Khandate, Gagan Wang, Boxuan Park, Sarah Ni, Weizhe Palacios, Joaquin Lampo, Kathyrn Wu, Philippe Ho, Rosh Chang, Eric Ciocarlie, Matei |
| contents | Pre-training on large datasets of robot demonstrations is a powerful technique for learning diverse manipulation skills but is often limited by the high cost and complexity of collecting robot-centric data, especially for tasks requiring tactile feedback. This work addresses these challenges by introducing a novel method for pre-training with multi-modal human demonstrations. Our approach jointly learns inverse and forward dynamics to extract latent state representations, towards learning manipulation specific representations. This enables efficient fine-tuning with only a small number of robot demonstrations, significantly improving data efficiency. Furthermore, our method allows for the use of multi-modal data, such as combination of vision and touch for manipulation. By leveraging latent dynamics modeling and tactile sensing, this approach paves the way for scalable robot manipulation learning based on human demonstrations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_12297 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Train Robots in a JIF: Joint Inverse and Forward Dynamics with Human and Robot Demonstrations Khandate, Gagan Wang, Boxuan Park, Sarah Ni, Weizhe Palacios, Joaquin Lampo, Kathyrn Wu, Philippe Ho, Rosh Chang, Eric Ciocarlie, Matei Robotics Pre-training on large datasets of robot demonstrations is a powerful technique for learning diverse manipulation skills but is often limited by the high cost and complexity of collecting robot-centric data, especially for tasks requiring tactile feedback. This work addresses these challenges by introducing a novel method for pre-training with multi-modal human demonstrations. Our approach jointly learns inverse and forward dynamics to extract latent state representations, towards learning manipulation specific representations. This enables efficient fine-tuning with only a small number of robot demonstrations, significantly improving data efficiency. Furthermore, our method allows for the use of multi-modal data, such as combination of vision and touch for manipulation. By leveraging latent dynamics modeling and tactile sensing, this approach paves the way for scalable robot manipulation learning based on human demonstrations. |
| title | Train Robots in a JIF: Joint Inverse and Forward Dynamics with Human and Robot Demonstrations |
| topic | Robotics |
| url | https://arxiv.org/abs/2503.12297 |