LFS: Learnable Frame Selector for Event-Aware and Temporally Diverse Video Captioning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917465939771392 |
|---|---|
| author | Chao, Lianying Yin, Linfeng Ren, Peiyu Jiang, Yifan Ren, Qiaoyu Shan, Dingcheng Pang, Jing-cheng Wu, Sijie Li, Xubin Zhang, Kai Chen, Xin |
| author_facet | Chao, Lianying Yin, Linfeng Ren, Peiyu Jiang, Yifan Ren, Qiaoyu Shan, Dingcheng Pang, Jing-cheng Wu, Sijie Li, Xubin Zhang, Kai Chen, Xin |
| contents | Video captioning models convert frames into visual tokens and generate descriptions with large language models (LLMs). Since encoding all frames is prohibitively expensive, uniform sampling is the default choice, but it enforces equal temporal coverage while ignoring the uneven events distribution. This motivates a Learnable Frame Selector (LFS) that selects temporally diverse and event-relevant frames. LFS explicitly models temporal importance to balance temporal diversity and event relevance, and employs a stratified strategy to ensure temporal coverage while avoiding clustering. Crucially, LFS leverages caption feedback from frozen video-LLMs to learn frame selection that directly optimizes downstream caption quality. Additionally, we identify the gap between existing benchmark and human's cognition. Thus, we introduce ICH-CC built from carefully designed questions by annotators that reflect human-consistent understanding of video. Experiments indicate that LFS consistently improves detailed video captioning across two representative community benchmarks and ICH-CC, achieving up to 2.0% gains on VDC and over 4% gains on ICH-CC. Moreover, we observe that enhanced captions with LFS leads to improved performance on video question answering. Overall, LFS provides an effective and easy-to-integrate solution for detailed video captioning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_14594 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | LFS: Learnable Frame Selector for Event-Aware and Temporally Diverse Video Captioning Chao, Lianying Yin, Linfeng Ren, Peiyu Jiang, Yifan Ren, Qiaoyu Shan, Dingcheng Pang, Jing-cheng Wu, Sijie Li, Xubin Zhang, Kai Chen, Xin Computer Vision and Pattern Recognition Video captioning models convert frames into visual tokens and generate descriptions with large language models (LLMs). Since encoding all frames is prohibitively expensive, uniform sampling is the default choice, but it enforces equal temporal coverage while ignoring the uneven events distribution. This motivates a Learnable Frame Selector (LFS) that selects temporally diverse and event-relevant frames. LFS explicitly models temporal importance to balance temporal diversity and event relevance, and employs a stratified strategy to ensure temporal coverage while avoiding clustering. Crucially, LFS leverages caption feedback from frozen video-LLMs to learn frame selection that directly optimizes downstream caption quality. Additionally, we identify the gap between existing benchmark and human's cognition. Thus, we introduce ICH-CC built from carefully designed questions by annotators that reflect human-consistent understanding of video. Experiments indicate that LFS consistently improves detailed video captioning across two representative community benchmarks and ICH-CC, achieving up to 2.0% gains on VDC and over 4% gains on ICH-CC. Moreover, we observe that enhanced captions with LFS leads to improved performance on video question answering. Overall, LFS provides an effective and easy-to-integrate solution for detailed video captioning. |
| title | LFS: Learnable Frame Selector for Event-Aware and Temporally Diverse Video Captioning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2601.14594 |