MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911602211553280 |
|---|---|
| author | Luo, Fuwen Lou, Shengfeng Chen, Chi Wang, Ziyue Li, Chenliang Shen, Weizhou Guo, Jiyue Li, Peng Yan, Ming Zhang, Ji Huang, Fei Liu, Yang |
| author_facet | Luo, Fuwen Lou, Shengfeng Chen, Chi Wang, Ziyue Li, Chenliang Shen, Weizhou Guo, Jiyue Li, Peng Yan, Ming Zhang, Ji Huang, Fei Liu, Yang |
| contents | Video temporal understanding is crucial for multimodal large language models (MLLMs) to reason over events in videos. Despite recent advances in general video understanding, current MLLMs still struggle with fine-grained temporal reasoning. While reinforcement learning (RL) has been explored to address this issue recently, existing RL approaches remain limited in performance on time-sensitive tasks. In this work, we propose MUSEG, a novel RL-based method that enhances temporal understanding by introducing timestamp-aware multi-segment grounding. MUSEG enables MLLMs to align queries with multiple relevant video segments, promoting more comprehensive temporal reasoning. To facilitate effective learning, we design a customized RL training recipe with phased rewards that progressively guides the model toward temporally grounded reasoning. Extensive experiments on temporal grounding and time-sensitive video question answering (QA) tasks demonstrate that MUSEG significantly outperforms existing methods and generalizes well across diverse temporal understanding scenarios. View our project at https://github.com/THUNLP-MT/MUSEG. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_20715 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding Luo, Fuwen Lou, Shengfeng Chen, Chi Wang, Ziyue Li, Chenliang Shen, Weizhou Guo, Jiyue Li, Peng Yan, Ming Zhang, Ji Huang, Fei Liu, Yang Computer Vision and Pattern Recognition Computation and Language Video temporal understanding is crucial for multimodal large language models (MLLMs) to reason over events in videos. Despite recent advances in general video understanding, current MLLMs still struggle with fine-grained temporal reasoning. While reinforcement learning (RL) has been explored to address this issue recently, existing RL approaches remain limited in performance on time-sensitive tasks. In this work, we propose MUSEG, a novel RL-based method that enhances temporal understanding by introducing timestamp-aware multi-segment grounding. MUSEG enables MLLMs to align queries with multiple relevant video segments, promoting more comprehensive temporal reasoning. To facilitate effective learning, we design a customized RL training recipe with phased rewards that progressively guides the model toward temporally grounded reasoning. Extensive experiments on temporal grounding and time-sensitive video question answering (QA) tasks demonstrate that MUSEG significantly outperforms existing methods and generalizes well across diverse temporal understanding scenarios. View our project at https://github.com/THUNLP-MT/MUSEG. |
| title | MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2505.20715 |