Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.05260 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915561979510784 |
|---|---|
| author | Lee, Junho Shin, Jeongwoo Ko, Seung Woo Ha, Seongsu Lee, Joonseok |
| author_facet | Lee, Junho Shin, Jeongwoo Ko, Seung Woo Ha, Seongsu Lee, Joonseok |
| contents | Given a video with $T$ frames, frame sampling is a task to select $N \ll T$ frames, so as to maximize the performance of a fixed video classifier. Not just brute-force search, but most existing methods suffer from its vast search space of $\binom{T}{N}$, especially when $N$ gets large. To address this challenge, we introduce a novel perspective of reducing the search space from $O(T^N)$ to $O(T)$. Instead of exploring the entire $O(T^N)$ space, our proposed semi-optimal policy selects the top $N$ frames based on the independently estimated value of each frame using per-frame confidence, significantly reducing the computational complexity. We verify that our semi-optimal policy can efficiently approximate the optimal policy, particularly under practical settings. Additionally, through extensive experiments on various datasets and model architectures, we demonstrate that learning our semi-optimal policy ensures stable and high performance regardless of the size of $N$ and $T$. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_05260 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Scalable Frame Sampling for Video Classification: A Semi-Optimal Policy Approach with Reduced Search Space Lee, Junho Shin, Jeongwoo Ko, Seung Woo Ha, Seongsu Lee, Joonseok Computer Vision and Pattern Recognition Given a video with $T$ frames, frame sampling is a task to select $N \ll T$ frames, so as to maximize the performance of a fixed video classifier. Not just brute-force search, but most existing methods suffer from its vast search space of $\binom{T}{N}$, especially when $N$ gets large. To address this challenge, we introduce a novel perspective of reducing the search space from $O(T^N)$ to $O(T)$. Instead of exploring the entire $O(T^N)$ space, our proposed semi-optimal policy selects the top $N$ frames based on the independently estimated value of each frame using per-frame confidence, significantly reducing the computational complexity. We verify that our semi-optimal policy can efficiently approximate the optimal policy, particularly under practical settings. Additionally, through extensive experiments on various datasets and model architectures, we demonstrate that learning our semi-optimal policy ensures stable and high performance regardless of the size of $N$ and $T$. |
| title | Scalable Frame Sampling for Video Classification: A Semi-Optimal Policy Approach with Reduced Search Space |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.05260 |