Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Tools
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908972149112832 |
|---|---|
| author | Yuan, Zhenlong Qu, Xiangyan Qian, Chengxuan Chen, Rui Tang, Jing Sun, Lei Chu, Xiangxiang Zhang, Dapeng Wang, Yiwei Cai, Yujun Li, Shuo |
| author_facet | Yuan, Zhenlong Qu, Xiangyan Qian, Chengxuan Chen, Rui Tang, Jing Sun, Lei Chu, Xiangxiang Zhang, Dapeng Wang, Yiwei Cai, Yujun Li, Shuo |
| contents | Multimodal large language models (MLLMs) have demonstrated remarkable potential in bridging visual and textual reasoning, yet their reliance on text-centric priors often limits their ability to disentangle semantically similar actions in open-vocabulary scenarios. To address this, we propose Video-STAR, a framework that harmonizes contextual sub-motion decomposition with tool-augmented reinforcement learning for open-vocabulary action recognition (OVAR). Unlike prior methods that treat actions as monolithic entities, our approach innovatively decomposes actions into discriminative sub-motions for fine-grained matching while dynamically invoking domain-specific tools for cross-modal interleaving, thereby enabling category-specific reasoning capacity and reducing cross-modal hallucination. Moreover, by designing a hierarchical reward that balances tool-usage efficiency, sub-motion relevance, and structural coherence in reasoning, our method autonomously leverages external tools to prioritize sub-motion patterns without explicit supervision, transmitting from text-centric reasoning to visually grounded inference. Extensive evaluations on HMDB-51, UCF-101, SSv2, Kinetics-400, and Kinetics-600 datasets demonstrate our state-of-the-art performance, outperforming existing methods in distinguishing fine-grained actions and handling cross-modal hallucination, validating our excellent robustness and generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_08480 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Tools Yuan, Zhenlong Qu, Xiangyan Qian, Chengxuan Chen, Rui Tang, Jing Sun, Lei Chu, Xiangxiang Zhang, Dapeng Wang, Yiwei Cai, Yujun Li, Shuo Computer Vision and Pattern Recognition Multimodal large language models (MLLMs) have demonstrated remarkable potential in bridging visual and textual reasoning, yet their reliance on text-centric priors often limits their ability to disentangle semantically similar actions in open-vocabulary scenarios. To address this, we propose Video-STAR, a framework that harmonizes contextual sub-motion decomposition with tool-augmented reinforcement learning for open-vocabulary action recognition (OVAR). Unlike prior methods that treat actions as monolithic entities, our approach innovatively decomposes actions into discriminative sub-motions for fine-grained matching while dynamically invoking domain-specific tools for cross-modal interleaving, thereby enabling category-specific reasoning capacity and reducing cross-modal hallucination. Moreover, by designing a hierarchical reward that balances tool-usage efficiency, sub-motion relevance, and structural coherence in reasoning, our method autonomously leverages external tools to prioritize sub-motion patterns without explicit supervision, transmitting from text-centric reasoning to visually grounded inference. Extensive evaluations on HMDB-51, UCF-101, SSv2, Kinetics-400, and Kinetics-600 datasets demonstrate our state-of-the-art performance, outperforming existing methods in distinguishing fine-grained actions and handling cross-modal hallucination, validating our excellent robustness and generalization. |
| title | Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Tools |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.08480 |