Elevating Skeleton-Based Action Recognition with Efficient Multi-Modality Self-Supervision
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909068211257344 |
|---|---|
| author | Wei, Yiping Peng, Kunyu Roitberg, Alina Zhang, Jiaming Zheng, Junwei Liu, Ruiping Chen, Yufan Yang, Kailun Stiefelhagen, Rainer |
| author_facet | Wei, Yiping Peng, Kunyu Roitberg, Alina Zhang, Jiaming Zheng, Junwei Liu, Ruiping Chen, Yufan Yang, Kailun Stiefelhagen, Rainer |
| contents | Self-supervised representation learning for human action recognition has developed rapidly in recent years. Most of the existing works are based on skeleton data while using a multi-modality setup. These works overlooked the differences in performance among modalities, which led to the propagation of erroneous knowledge between modalities while only three fundamental modalities, i.e., joints, bones, and motions are used, hence no additional modalities are explored.
In this work, we first propose an Implicit Knowledge Exchange Module (IKEM) which alleviates the propagation of erroneous knowledge between low-performance modalities. Then, we further propose three new modalities to enrich the complementary information between modalities. Finally, to maintain efficiency when introducing new modalities, we propose a novel teacher-student framework to distill the knowledge from the secondary modalities into the mandatory modalities considering the relationship constrained by anchors, positives, and negatives, named relational cross-modality knowledge distillation. The experimental results demonstrate the effectiveness of our approach, unlocking the efficient use of skeleton-based multi-modality data. Source code will be made publicly available at https://github.com/desehuileng0o0/IKEM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_12009 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Elevating Skeleton-Based Action Recognition with Efficient Multi-Modality Self-Supervision Wei, Yiping Peng, Kunyu Roitberg, Alina Zhang, Jiaming Zheng, Junwei Liu, Ruiping Chen, Yufan Yang, Kailun Stiefelhagen, Rainer Computer Vision and Pattern Recognition Multimedia Robotics Image and Video Processing Self-supervised representation learning for human action recognition has developed rapidly in recent years. Most of the existing works are based on skeleton data while using a multi-modality setup. These works overlooked the differences in performance among modalities, which led to the propagation of erroneous knowledge between modalities while only three fundamental modalities, i.e., joints, bones, and motions are used, hence no additional modalities are explored. In this work, we first propose an Implicit Knowledge Exchange Module (IKEM) which alleviates the propagation of erroneous knowledge between low-performance modalities. Then, we further propose three new modalities to enrich the complementary information between modalities. Finally, to maintain efficiency when introducing new modalities, we propose a novel teacher-student framework to distill the knowledge from the secondary modalities into the mandatory modalities considering the relationship constrained by anchors, positives, and negatives, named relational cross-modality knowledge distillation. The experimental results demonstrate the effectiveness of our approach, unlocking the efficient use of skeleton-based multi-modality data. Source code will be made publicly available at https://github.com/desehuileng0o0/IKEM. |
| title | Elevating Skeleton-Based Action Recognition with Efficient Multi-Modality Self-Supervision |
| topic | Computer Vision and Pattern Recognition Multimedia Robotics Image and Video Processing |
| url | https://arxiv.org/abs/2309.12009 |