Towards Fusing Point Cloud and Visual Representations for Imitation Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913698038153216 |
|---|---|
| author | Donat, Atalay Jia, Xiaogang Huang, Xi Taranovic, Aleksandar Blessing, Denis Li, Ge Zhou, Hongyi Zhang, Hanyi Lioutikov, Rudolf Neumann, Gerhard |
| author_facet | Donat, Atalay Jia, Xiaogang Huang, Xi Taranovic, Aleksandar Blessing, Denis Li, Ge Zhou, Hongyi Zhang, Hanyi Lioutikov, Rudolf Neumann, Gerhard |
| contents | Learning for manipulation requires using policies that have access to rich sensory information such as point clouds or RGB images. Point clouds efficiently capture geometric structures, making them essential for manipulation tasks in imitation learning. In contrast, RGB images provide rich texture and semantic information that can be crucial for certain tasks. Existing approaches for fusing both modalities assign 2D image features to point clouds. However, such approaches often lose global contextual information from the original images. In this work, we propose FPV-Net, a novel imitation learning method that effectively combines the strengths of both point cloud and RGB modalities. Our method conditions the point-cloud encoder on global and local image tokens using adaptive layer norm conditioning, leveraging the beneficial properties of both modalities. Through extensive experiments on the challenging RoboCasa benchmark, we demonstrate the limitations of relying on either modality alone and show that our method achieves state-of-the-art performance across all tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_12320 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Fusing Point Cloud and Visual Representations for Imitation Learning Donat, Atalay Jia, Xiaogang Huang, Xi Taranovic, Aleksandar Blessing, Denis Li, Ge Zhou, Hongyi Zhang, Hanyi Lioutikov, Rudolf Neumann, Gerhard Robotics Computer Vision and Pattern Recognition Learning for manipulation requires using policies that have access to rich sensory information such as point clouds or RGB images. Point clouds efficiently capture geometric structures, making them essential for manipulation tasks in imitation learning. In contrast, RGB images provide rich texture and semantic information that can be crucial for certain tasks. Existing approaches for fusing both modalities assign 2D image features to point clouds. However, such approaches often lose global contextual information from the original images. In this work, we propose FPV-Net, a novel imitation learning method that effectively combines the strengths of both point cloud and RGB modalities. Our method conditions the point-cloud encoder on global and local image tokens using adaptive layer norm conditioning, leveraging the beneficial properties of both modalities. Through extensive experiments on the challenging RoboCasa benchmark, we demonstrate the limitations of relying on either modality alone and show that our method achieves state-of-the-art performance across all tasks. |
| title | Towards Fusing Point Cloud and Visual Representations for Imitation Learning |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.12320 |