Making Every Frame Matter: Continuous Activity Recognition in Streaming Video via Adaptive Video Context Modeling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910873445990400 |
|---|---|
| author | Wu, Hao Bai, Donglin Jiang, Shiqi Zhang, Qianxi Yang, Yifan Ding, Xin Cao, Ting Liu, Yunxin Xu, Fengyuan |
| author_facet | Wu, Hao Bai, Donglin Jiang, Shiqi Zhang, Qianxi Yang, Yifan Ding, Xin Cao, Ting Liu, Yunxin Xu, Fengyuan |
| contents | Video activity recognition has become increasingly important in robots and embodied AI. Recognizing continuous video activities poses considerable challenges due to the fast expansion of streaming video, which contains multi-scale and untrimmed activities. We introduce a novel system, CARS, to overcome these issues through adaptive video context modeling. Adaptive video context modeling refers to selectively maintaining activity-related features in temporal and spatial dimensions. CARS has two key designs. The first is an activity spatial feature extraction by eliminating irrelevant visual features while maintaining recognition accuracy. The second is an activity-aware state update introducing dynamic adaptability to better preserve the video context for multi-scale activity recognition. Our CARS runs at speeds $>$30 FPS on typical edge devices and outperforms all baselines by 1.2\% to 79.7\% in accuracy. Moreover, we explore applying CARS to a large video model as a video encoder. Experimental results show that our CARS can result in a 0.46-point enhancement (on a 5-point scale) on the in-distribution video activity dataset, and an improvement ranging from 1.19\% to 4\% on zero-shot video activity datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_14993 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Making Every Frame Matter: Continuous Activity Recognition in Streaming Video via Adaptive Video Context Modeling Wu, Hao Bai, Donglin Jiang, Shiqi Zhang, Qianxi Yang, Yifan Ding, Xin Cao, Ting Liu, Yunxin Xu, Fengyuan Computer Vision and Pattern Recognition Video activity recognition has become increasingly important in robots and embodied AI. Recognizing continuous video activities poses considerable challenges due to the fast expansion of streaming video, which contains multi-scale and untrimmed activities. We introduce a novel system, CARS, to overcome these issues through adaptive video context modeling. Adaptive video context modeling refers to selectively maintaining activity-related features in temporal and spatial dimensions. CARS has two key designs. The first is an activity spatial feature extraction by eliminating irrelevant visual features while maintaining recognition accuracy. The second is an activity-aware state update introducing dynamic adaptability to better preserve the video context for multi-scale activity recognition. Our CARS runs at speeds $>$30 FPS on typical edge devices and outperforms all baselines by 1.2\% to 79.7\% in accuracy. Moreover, we explore applying CARS to a large video model as a video encoder. Experimental results show that our CARS can result in a 0.46-point enhancement (on a 5-point scale) on the in-distribution video activity dataset, and an improvement ranging from 1.19\% to 4\% on zero-shot video activity datasets. |
| title | Making Every Frame Matter: Continuous Activity Recognition in Streaming Video via Adaptive Video Context Modeling |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.14993 |