IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909394457853952 |
|---|---|
| author | Liu, Yunong Eyzaguirre, Cristobal Li, Manling Khanna, Shubh Niebles, Juan Carlos Ravi, Vineeth Mishra, Saumitra Liu, Weiyu Wu, Jiajun |
| author_facet | Liu, Yunong Eyzaguirre, Cristobal Li, Manling Khanna, Shubh Niebles, Juan Carlos Ravi, Vineeth Mishra, Saumitra Liu, Weiyu Wu, Jiajun |
| contents | Shape assembly is a ubiquitous task in daily life, integral for constructing complex 3D structures like IKEA furniture. While significant progress has been made in developing autonomous agents for shape assembly, existing datasets have not yet tackled the 4D grounding of assembly instructions in videos, essential for a holistic understanding of assembly in 3D space over time. We introduce IKEA Video Manuals, a dataset that features 3D models of furniture parts, instructional manuals, assembly videos from the Internet, and most importantly, annotations of dense spatio-temporal alignments between these data modalities. To demonstrate the utility of IKEA Video Manuals, we present five applications essential for shape assembly: assembly plan generation, part-conditioned segmentation, part-conditioned pose estimation, video object segmentation, and furniture assembly based on instructional video manuals. For each application, we provide evaluation metrics and baseline methods. Through experiments on our annotated data, we highlight many challenges in grounding assembly instructions in videos to improve shape assembly, including handling occlusions, varying viewpoints, and extended assembly sequences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_11409 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos Liu, Yunong Eyzaguirre, Cristobal Li, Manling Khanna, Shubh Niebles, Juan Carlos Ravi, Vineeth Mishra, Saumitra Liu, Weiyu Wu, Jiajun Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Robotics Shape assembly is a ubiquitous task in daily life, integral for constructing complex 3D structures like IKEA furniture. While significant progress has been made in developing autonomous agents for shape assembly, existing datasets have not yet tackled the 4D grounding of assembly instructions in videos, essential for a holistic understanding of assembly in 3D space over time. We introduce IKEA Video Manuals, a dataset that features 3D models of furniture parts, instructional manuals, assembly videos from the Internet, and most importantly, annotations of dense spatio-temporal alignments between these data modalities. To demonstrate the utility of IKEA Video Manuals, we present five applications essential for shape assembly: assembly plan generation, part-conditioned segmentation, part-conditioned pose estimation, video object segmentation, and furniture assembly based on instructional video manuals. For each application, we provide evaluation metrics and baseline methods. Through experiments on our annotated data, we highlight many challenges in grounding assembly instructions in videos to improve shape assembly, including handling occlusions, varying viewpoints, and extended assembly sequences. |
| title | IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Robotics |
| url | https://arxiv.org/abs/2411.11409 |