Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.08646 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908289522991104 |
|---|---|
| author | Liu, Jihao Yu, Zhiding Lan, Shiyi Wang, Shihao Fang, Rongyao Kautz, Jan Li, Hongsheng Alvare, Jose M. |
| author_facet | Liu, Jihao Yu, Zhiding Lan, Shiyi Wang, Shihao Fang, Rongyao Kautz, Jan Li, Hongsheng Alvare, Jose M. |
| contents | This paper presents StreamChat, a novel approach that enhances the interaction capabilities of Large Multimodal Models (LMMs) with streaming video content. In streaming interaction scenarios, existing methods rely solely on visual information available at the moment a question is posed, resulting in significant delays as the model remains unaware of subsequent changes in the streaming video. StreamChat addresses this limitation by innovatively updating the visual context at each decoding step, ensuring that the model utilizes up-to-date video content throughout the decoding process. Additionally, we introduce a flexible and efficient crossattention-based architecture to process dynamic streaming inputs while maintaining inference efficiency for streaming interactions. Furthermore, we construct a new dense instruction dataset to facilitate the training of streaming interaction models, complemented by a parallel 3D-RoPE mechanism that encodes the relative temporal information of visual and text tokens. Experimental results demonstrate that StreamChat achieves competitive performance on established image and video benchmarks and exhibits superior capabilities in streaming interaction scenarios compared to state-of-the-art video LMM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_08646 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | StreamChat: Chatting with Streaming Video Liu, Jihao Yu, Zhiding Lan, Shiyi Wang, Shihao Fang, Rongyao Kautz, Jan Li, Hongsheng Alvare, Jose M. Computer Vision and Pattern Recognition This paper presents StreamChat, a novel approach that enhances the interaction capabilities of Large Multimodal Models (LMMs) with streaming video content. In streaming interaction scenarios, existing methods rely solely on visual information available at the moment a question is posed, resulting in significant delays as the model remains unaware of subsequent changes in the streaming video. StreamChat addresses this limitation by innovatively updating the visual context at each decoding step, ensuring that the model utilizes up-to-date video content throughout the decoding process. Additionally, we introduce a flexible and efficient crossattention-based architecture to process dynamic streaming inputs while maintaining inference efficiency for streaming interactions. Furthermore, we construct a new dense instruction dataset to facilitate the training of streaming interaction models, complemented by a parallel 3D-RoPE mechanism that encodes the relative temporal information of visual and text tokens. Experimental results demonstrate that StreamChat achieves competitive performance on established image and video benchmarks and exhibits superior capabilities in streaming interaction scenarios compared to state-of-the-art video LMM. |
| title | StreamChat: Chatting with Streaming Video |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.08646 |