SUGAR: Learning Skeleton Representation with Visual-Motion Knowledge for Action Recognition
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914156154716160 |
|---|---|
| author | Ye, Qilang Zhou, Yu He, Lian Zhang, Jie Guo, Xuanming Zhang, Jiayu Tan, Mingkui Xie, Weicheng Sun, Yue Tan, Tao Yuan, Xiaochen Khoriba, Ghada Yu, Zitong |
| author_facet | Ye, Qilang Zhou, Yu He, Lian Zhang, Jie Guo, Xuanming Zhang, Jiayu Tan, Mingkui Xie, Weicheng Sun, Yue Tan, Tao Yuan, Xiaochen Khoriba, Ghada Yu, Zitong |
| contents | Large Language Models (LLMs) hold rich implicit knowledge and powerful transferability. In this paper, we explore the combination of LLMs with the human skeleton to perform action classification and description. However, when treating LLM as a recognizer, two questions arise: 1) How can LLMs understand skeleton? 2) How can LLMs distinguish among actions? To address these problems, we introduce a novel paradigm named learning Skeleton representation with visUal-motion knowledGe for Action Recognition (SUGAR). In our pipeline, we first utilize off-the-shelf large-scale video models as a knowledge base to generate visual, motion information related to actions. Then, we propose to supervise skeleton learning through this prior knowledge to yield discrete representations. Finally, we use the LLM with untouched pre-training weights to understand these representations and generate the desired action targets and descriptions. Notably, we present a Temporal Query Projection (TQP) module to continuously model the skeleton signals with long sequences. Experiments on several skeleton-based action classification benchmarks demonstrate the efficacy of our SUGAR. Moreover, experiments on zero-shot scenarios show that SUGAR is more versatile than linear-based methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_10091 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SUGAR: Learning Skeleton Representation with Visual-Motion Knowledge for Action Recognition Ye, Qilang Zhou, Yu He, Lian Zhang, Jie Guo, Xuanming Zhang, Jiayu Tan, Mingkui Xie, Weicheng Sun, Yue Tan, Tao Yuan, Xiaochen Khoriba, Ghada Yu, Zitong Computer Vision and Pattern Recognition Large Language Models (LLMs) hold rich implicit knowledge and powerful transferability. In this paper, we explore the combination of LLMs with the human skeleton to perform action classification and description. However, when treating LLM as a recognizer, two questions arise: 1) How can LLMs understand skeleton? 2) How can LLMs distinguish among actions? To address these problems, we introduce a novel paradigm named learning Skeleton representation with visUal-motion knowledGe for Action Recognition (SUGAR). In our pipeline, we first utilize off-the-shelf large-scale video models as a knowledge base to generate visual, motion information related to actions. Then, we propose to supervise skeleton learning through this prior knowledge to yield discrete representations. Finally, we use the LLM with untouched pre-training weights to understand these representations and generate the desired action targets and descriptions. Notably, we present a Temporal Query Projection (TQP) module to continuously model the skeleton signals with long sequences. Experiments on several skeleton-based action classification benchmarks demonstrate the efficacy of our SUGAR. Moreover, experiments on zero-shot scenarios show that SUGAR is more versatile than linear-based methods. |
| title | SUGAR: Learning Skeleton Representation with Visual-Motion Knowledge for Action Recognition |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.10091 |