TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916174297563136 |
|---|---|
| author | Liu, Yun Yang, Haolin Si, Xu Liu, Ling Li, Zipeng Zhang, Yuxiang Liu, Yebin Yi, Li |
| author_facet | Liu, Yun Yang, Haolin Si, Xu Liu, Ling Li, Zipeng Zhang, Yuxiang Liu, Yebin Yi, Li |
| contents | Humans commonly work with multiple objects in daily life and can intuitively transfer manipulation skills to novel objects by understanding object functional regularities. However, existing technical approaches for analyzing and synthesizing hand-object manipulation are mostly limited to handling a single hand and object due to the lack of data support. To address this, we construct TACO, an extensive bimanual hand-object-interaction dataset spanning a large variety of tool-action-object compositions for daily human activities. TACO contains 2.5K motion sequences paired with third-person and egocentric views, precise hand-object 3D meshes, and action labels. To rapidly expand the data scale, we present a fully automatic data acquisition pipeline combining multi-view sensing with an optical motion capture system. With the vast research fields provided by TACO, we benchmark three generalizable hand-object-interaction tasks: compositional action recognition, generalizable hand-object motion forecasting, and cooperative grasp synthesis. Extensive experiments reveal new insights, challenges, and opportunities for advancing the studies of generalizable hand-object motion analysis and synthesis. Our data and code are available at https://taco2024.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_08399 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding Liu, Yun Yang, Haolin Si, Xu Liu, Ling Li, Zipeng Zhang, Yuxiang Liu, Yebin Yi, Li Computer Vision and Pattern Recognition Humans commonly work with multiple objects in daily life and can intuitively transfer manipulation skills to novel objects by understanding object functional regularities. However, existing technical approaches for analyzing and synthesizing hand-object manipulation are mostly limited to handling a single hand and object due to the lack of data support. To address this, we construct TACO, an extensive bimanual hand-object-interaction dataset spanning a large variety of tool-action-object compositions for daily human activities. TACO contains 2.5K motion sequences paired with third-person and egocentric views, precise hand-object 3D meshes, and action labels. To rapidly expand the data scale, we present a fully automatic data acquisition pipeline combining multi-view sensing with an optical motion capture system. With the vast research fields provided by TACO, we benchmark three generalizable hand-object-interaction tasks: compositional action recognition, generalizable hand-object motion forecasting, and cooperative grasp synthesis. Extensive experiments reveal new insights, challenges, and opportunities for advancing the studies of generalizable hand-object motion analysis and synthesis. Our data and code are available at https://taco2024.github.io. |
| title | TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.08399 |