Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.10976 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912644280090624 |
|---|---|
| author | Wang, Wentao Zou, Heqing Luo, Tianze Huang, Rui Zhao, Yutian Wang, Zhuochen Zhang, Hansheng Qin, Chengwei Wang, Yan Zhao, Lin Zhang, Huaijian |
| author_facet | Wang, Wentao Zou, Heqing Luo, Tianze Huang, Rui Zhao, Yutian Wang, Zhuochen Zhang, Hansheng Qin, Chengwei Wang, Yan Zhao, Lin Zhang, Huaijian |
| contents | Recent progress in Multimodal Large Language Models (MLLMs) has demonstrated strong semantic understanding capabilities, but struggles to perform precise spatio-temporal understanding. Existing spatio-temporal methods primarily focus on the video itself, while overlooking the physical information within the video, such as multi-object layouts and motion. Such limitations restrict the use of MLLMs in downstream applications that demand high precision, including embodied intelligence and VR. To address this issue, we present Video-STR, a novel graph-based reinforcement method for precise Video Spatio-Temporal Reasoning. Building upon the capacity of Reinforcement Learning with Verifiable Reward (RLVR) to improve model abilities, we introduce a reasoning mechanism using graph-based Group Relative Policy Optimization (GRPO) method to guide the model in inferring the underlying spatio-temporal topology of scenarios during the thinking process. To resolve the lack of spatio-temporal training data, we construct the STV-205k dataset with 205k question-answering pairs, covering dynamic multi-object scenes in both indoor and outdoor environments, to support the model training. Experiments show that Video-STR achieves state-of-the-art results on various benchmarks, outperforming the base model by 13% on STI-Bench, and demonstrating the effectiveness of our approach and dataset. Code, model, and data will be released. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_10976 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Video-STR: Reinforcing MLLMs in Video Spatio-Temporal Reasoning with Relation Graph Wang, Wentao Zou, Heqing Luo, Tianze Huang, Rui Zhao, Yutian Wang, Zhuochen Zhang, Hansheng Qin, Chengwei Wang, Yan Zhao, Lin Zhang, Huaijian Artificial Intelligence 68T05 I.2.10 Recent progress in Multimodal Large Language Models (MLLMs) has demonstrated strong semantic understanding capabilities, but struggles to perform precise spatio-temporal understanding. Existing spatio-temporal methods primarily focus on the video itself, while overlooking the physical information within the video, such as multi-object layouts and motion. Such limitations restrict the use of MLLMs in downstream applications that demand high precision, including embodied intelligence and VR. To address this issue, we present Video-STR, a novel graph-based reinforcement method for precise Video Spatio-Temporal Reasoning. Building upon the capacity of Reinforcement Learning with Verifiable Reward (RLVR) to improve model abilities, we introduce a reasoning mechanism using graph-based Group Relative Policy Optimization (GRPO) method to guide the model in inferring the underlying spatio-temporal topology of scenarios during the thinking process. To resolve the lack of spatio-temporal training data, we construct the STV-205k dataset with 205k question-answering pairs, covering dynamic multi-object scenes in both indoor and outdoor environments, to support the model training. Experiments show that Video-STR achieves state-of-the-art results on various benchmarks, outperforming the base model by 13% on STI-Bench, and demonstrating the effectiveness of our approach and dataset. Code, model, and data will be released. |
| title | Video-STR: Reinforcing MLLMs in Video Spatio-Temporal Reasoning with Relation Graph |
| topic | Artificial Intelligence 68T05 I.2.10 |
| url | https://arxiv.org/abs/2510.10976 |