CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914460950593536 |
|---|---|
| author | Zou, Yulin Chen, Yan Chen, Wenyan Park, JooYoung Nitin, Shivaraman Tao, Luo Romero, Francisco Ustiugov, Dmitrii |
| author_facet | Zou, Yulin Chen, Yan Chen, Wenyan Park, JooYoung Nitin, Shivaraman Tao, Luo Romero, Francisco Ustiugov, Dmitrii |
| contents | Video streaming analytics is a crucial workload for vision-language model serving, but the high cost of multimodal inference limits scalability. Prior systems reduce inference cost by exploiting temporal and spatial redundancy in video streams, but they target either the vision transformer (ViT) or the LLM with a limited view, leaving end-to-end opportunities untapped. Moreover, existing methods incur significant overhead to identify redundancy, either through offline profiling and training or costly online computation, making them ill-suited for dynamic real-time streams.
We present CodecSight, a codec-guided streaming video analytics system, built on a key observation that video codecs already extract the temporal and spatial structure of each stream as a byproduct of compression. CodecSight treats this codec metadata as a low-cost runtime signal to unify optimization across video decoding, visual processing, and LLM prefilling, with transmission reduction as an inherent benefit of operating directly on compressed bitstreams. This drives codec-guided patch pruning before ViT encoding and selective key-value cache refresh during LLM prefilling, both of which are fully online and do not require offline training. Experiments show that CodecSight achieves an improvement in throughput of up to 3$\times$, and a reduction of up to 87% in GPU compute over state-of-the-art baselines, maintaining competitive accuracy with only 0$\sim$8% F1 drop. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_06036 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference Zou, Yulin Chen, Yan Chen, Wenyan Park, JooYoung Nitin, Shivaraman Tao, Luo Romero, Francisco Ustiugov, Dmitrii Distributed, Parallel, and Cluster Computing Computer Vision and Pattern Recognition Machine Learning Video streaming analytics is a crucial workload for vision-language model serving, but the high cost of multimodal inference limits scalability. Prior systems reduce inference cost by exploiting temporal and spatial redundancy in video streams, but they target either the vision transformer (ViT) or the LLM with a limited view, leaving end-to-end opportunities untapped. Moreover, existing methods incur significant overhead to identify redundancy, either through offline profiling and training or costly online computation, making them ill-suited for dynamic real-time streams. We present CodecSight, a codec-guided streaming video analytics system, built on a key observation that video codecs already extract the temporal and spatial structure of each stream as a byproduct of compression. CodecSight treats this codec metadata as a low-cost runtime signal to unify optimization across video decoding, visual processing, and LLM prefilling, with transmission reduction as an inherent benefit of operating directly on compressed bitstreams. This drives codec-guided patch pruning before ViT encoding and selective key-value cache refresh during LLM prefilling, both of which are fully online and do not require offline training. Experiments show that CodecSight achieves an improvement in throughput of up to 3$\times$, and a reduction of up to 87% in GPU compute over state-of-the-art baselines, maintaining competitive accuracy with only 0$\sim$8% F1 drop. |
| title | CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference |
| topic | Distributed, Parallel, and Cluster Computing Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2604.06036 |