Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2304.00553 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909158260867072 |
|---|---|
| author | Li, Yong-Lu Wu, Xiaoqian Liu, Xinpeng Wang, Zehao Dou, Yiming Ji, Yikun Zhang, Junyi Li, Yixing Tan, Jingru Lu, Xudong Lu, Cewu |
| author_facet | Li, Yong-Lu Wu, Xiaoqian Liu, Xinpeng Wang, Zehao Dou, Yiming Ji, Yikun Zhang, Junyi Li, Yixing Tan, Jingru Lu, Xudong Lu, Cewu |
| contents | Action understanding has attracted long-term attention. It can be formed as the mapping from the physical space to the semantic space. Typically, researchers built datasets according to idiosyncratic choices to define classes and push the envelope of benchmarks respectively. Datasets are incompatible with each other like "Isolated Islands" due to semantic gaps and various class granularities, e.g., do housework in dataset A and wash plate in dataset B. We argue that we need a more principled semantic space to concentrate the community efforts and use all datasets together to pursue generalizable action learning. To this end, we design a structured action semantic space given verb taxonomy hierarchy and covering massive actions. By aligning the classes of previous datasets to our semantic space, we gather (image/video/skeleton/MoCap) datasets into a unified database in a unified label system, i.e., bridging "isolated islands" into a "Pangea". Accordingly, we propose a novel model mapping from the physical space to semantic space to fully use Pangea. In extensive experiments, our new system shows significant superiority, especially in transfer learning. Our code and data will be made public at https://mvig-rhos.com/pangea. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_00553 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding Li, Yong-Lu Wu, Xiaoqian Liu, Xinpeng Wang, Zehao Dou, Yiming Ji, Yikun Zhang, Junyi Li, Yixing Tan, Jingru Lu, Xudong Lu, Cewu Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Action understanding has attracted long-term attention. It can be formed as the mapping from the physical space to the semantic space. Typically, researchers built datasets according to idiosyncratic choices to define classes and push the envelope of benchmarks respectively. Datasets are incompatible with each other like "Isolated Islands" due to semantic gaps and various class granularities, e.g., do housework in dataset A and wash plate in dataset B. We argue that we need a more principled semantic space to concentrate the community efforts and use all datasets together to pursue generalizable action learning. To this end, we design a structured action semantic space given verb taxonomy hierarchy and covering massive actions. By aligning the classes of previous datasets to our semantic space, we gather (image/video/skeleton/MoCap) datasets into a unified database in a unified label system, i.e., bridging "isolated islands" into a "Pangea". Accordingly, we propose a novel model mapping from the physical space to semantic space to fully use Pangea. In extensive experiments, our new system shows significant superiority, especially in transfer learning. Our code and data will be made public at https://mvig-rhos.com/pangea. |
| title | From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2304.00553 |