Configurable Embodied Data Generation for Class-Agnostic RGB-D Video Segmentation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912075108843520 |
|---|---|
| author | Opipari, Anthony Krishnan, Aravindhan K Gayaka, Shreekant Sun, Min Kuo, Cheng-Hao Sen, Arnie Jenkins, Odest Chadwicke |
| author_facet | Opipari, Anthony Krishnan, Aravindhan K Gayaka, Shreekant Sun, Min Kuo, Cheng-Hao Sen, Arnie Jenkins, Odest Chadwicke |
| contents | This paper presents a method for generating large-scale datasets to improve class-agnostic video segmentation across robots with different form factors. Specifically, we consider the question of whether video segmentation models trained on generic segmentation data could be more effective for particular robot platforms if robot embodiment is factored into the data generation process. To answer this question, a pipeline is formulated for using 3D reconstructions (e.g. from HM3DSem) to generate segmented videos that are configurable based on a robot's embodiment (e.g. sensor type, sensor placement, and illumination source). A resulting massive RGB-D video panoptic segmentation dataset (MVPd) is introduced for extensive benchmarking with foundation and video segmentation models, as well as to support embodiment-focused research in video segmentation. Our experimental findings demonstrate that using MVPd for finetuning can lead to performance improvements when transferring foundation models to certain robot embodiments, such as specific camera placements. These experiments also show that using 3D modalities (depth images and camera pose) can lead to improvements in video segmentation accuracy and consistency. The project webpage is available at https://topipari.com/projects/MVPd |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12995 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Configurable Embodied Data Generation for Class-Agnostic RGB-D Video Segmentation Opipari, Anthony Krishnan, Aravindhan K Gayaka, Shreekant Sun, Min Kuo, Cheng-Hao Sen, Arnie Jenkins, Odest Chadwicke Robotics Computer Vision and Pattern Recognition This paper presents a method for generating large-scale datasets to improve class-agnostic video segmentation across robots with different form factors. Specifically, we consider the question of whether video segmentation models trained on generic segmentation data could be more effective for particular robot platforms if robot embodiment is factored into the data generation process. To answer this question, a pipeline is formulated for using 3D reconstructions (e.g. from HM3DSem) to generate segmented videos that are configurable based on a robot's embodiment (e.g. sensor type, sensor placement, and illumination source). A resulting massive RGB-D video panoptic segmentation dataset (MVPd) is introduced for extensive benchmarking with foundation and video segmentation models, as well as to support embodiment-focused research in video segmentation. Our experimental findings demonstrate that using MVPd for finetuning can lead to performance improvements when transferring foundation models to certain robot embodiments, such as specific camera placements. These experiments also show that using 3D modalities (depth images and camera pose) can lead to improvements in video segmentation accuracy and consistency. The project webpage is available at https://topipari.com/projects/MVPd |
| title | Configurable Embodied Data Generation for Class-Agnostic RGB-D Video Segmentation |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.12995 |