Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.08326 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908277382578176 |
|---|---|
| author | Heo, Miran Chen, Min-Hung Huang, De-An Liu, Sifei Radhakrishnan, Subhashree Kim, Seon Joo Wang, Yu-Chiang Frank Hachiuma, Ryo |
| author_facet | Heo, Miran Chen, Min-Hung Huang, De-An Liu, Sifei Radhakrishnan, Subhashree Kim, Seon Joo Wang, Yu-Chiang Frank Hachiuma, Ryo |
| contents | We present Omni-RGPT, a multimodal large language model designed to facilitate region-level comprehension for both images and videos. To achieve consistent region representation across spatio-temporal dimensions, we introduce Token Mark, a set of tokens highlighting the target regions within the visual feature space. These tokens are directly embedded into spatial regions using region prompts (e.g., boxes or masks) and simultaneously incorporated into the text prompt to specify the target, establishing a direct connection between visual and text tokens. To further support robust video understanding without requiring tracklets, we introduce an auxiliary task that guides Token Mark by leveraging the consistency of the tokens, enabling stable region interpretation across the video. Additionally, we introduce a large-scale region-level video instruction dataset (RegVID-300k). Omni-RGPT achieves state-of-the-art results on image and video-based commonsense reasoning benchmarks while showing strong performance in captioning and referring expression comprehension tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_08326 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks Heo, Miran Chen, Min-Hung Huang, De-An Liu, Sifei Radhakrishnan, Subhashree Kim, Seon Joo Wang, Yu-Chiang Frank Hachiuma, Ryo Computer Vision and Pattern Recognition We present Omni-RGPT, a multimodal large language model designed to facilitate region-level comprehension for both images and videos. To achieve consistent region representation across spatio-temporal dimensions, we introduce Token Mark, a set of tokens highlighting the target regions within the visual feature space. These tokens are directly embedded into spatial regions using region prompts (e.g., boxes or masks) and simultaneously incorporated into the text prompt to specify the target, establishing a direct connection between visual and text tokens. To further support robust video understanding without requiring tracklets, we introduce an auxiliary task that guides Token Mark by leveraging the consistency of the tokens, enabling stable region interpretation across the video. Additionally, we introduce a large-scale region-level video instruction dataset (RegVID-300k). Omni-RGPT achieves state-of-the-art results on image and video-based commonsense reasoning benchmarks while showing strong performance in captioning and referring expression comprehension tasks. |
| title | Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.08326 |