T*: Re-thinking Temporal Search for Long-Form Video Understanding

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ye, Jinhui, Wang, Zihan, Sun, Haosen, Chandrasegaran, Keshigeyan, Durante, Zane, Eyzaguirre, Cristobal, Bisk, Yonatan, Niebles, Juan Carlos, Adeli, Ehsan, Fei-Fei, Li, Wu, Jiajun, Li, Manling
Format:	Preprint
Published:	2025
Subjects:	Computer Vision and Pattern Recognition
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866912550863503360
author	Ye, Jinhui Wang, Zihan Sun, Haosen Chandrasegaran, Keshigeyan Durante, Zane Eyzaguirre, Cristobal Bisk, Yonatan Niebles, Juan Carlos Adeli, Ehsan Fei-Fei, Li Wu, Jiajun Li, Manling
author_facet	Ye, Jinhui Wang, Zihan Sun, Haosen Chandrasegaran, Keshigeyan Durante, Zane Eyzaguirre, Cristobal Bisk, Yonatan Niebles, Juan Carlos Adeli, Ehsan Fei-Fei, Li Wu, Jiajun Li, Manling
contents	Efficiently understanding long-form videos remains a significant challenge in computer vision. In this work, we revisit temporal search paradigms for long-form video understanding and address a fundamental issue pertaining to all state-of-the-art (SOTA) long-context vision-language models (VLMs). Our contributions are twofold: First, we frame temporal search as a Long Video Haystack problem: finding a minimal set of relevant frames (e.g., one to five) from tens of thousands based on specific queries. Upon this formulation, we introduce LV-Haystack, the first dataset with 480 hours of videos, 15,092 human-annotated instances for both training and evaluation aiming to improve temporal search quality and efficiency. Results on LV-Haystack highlight a significant research gap in temporal search capabilities, with current SOTA search methods only achieving 2.1% temporal F1 score on the Longvideobench subset. Next, inspired by visual search in images, we propose a lightweight temporal search framework, T* that reframes costly temporal search as spatial search. T* leverages powerful visual localization techniques commonly used in images and introduces an adaptive zooming-in mechanism that operates across both temporal and spatial dimensions. Extensive experiments show that integrating T* with existing methods significantly improves SOTA long-form video understanding. Under an inference budget of 32 frames, T* improves GPT-4o's performance from 50.5% to 53.1% and LLaVA-OneVision-OV-72B's performance from 56.5% to 62.4% on the Longvideobench XL subset. Our code, benchmark, and models are provided in the Supplementary material.
format	Preprint
id	arxiv_https___arxiv_org_abs_2504_02259
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	T: Re-thinking Temporal Search for Long-Form Video Understanding Ye, Jinhui Wang, Zihan Sun, Haosen Chandrasegaran, Keshigeyan Durante, Zane Eyzaguirre, Cristobal Bisk, Yonatan Niebles, Juan Carlos Adeli, Ehsan Fei-Fei, Li Wu, Jiajun Li, Manling Computer Vision and Pattern Recognition Efficiently understanding long-form videos remains a significant challenge in computer vision. In this work, we revisit temporal search paradigms for long-form video understanding and address a fundamental issue pertaining to all state-of-the-art (SOTA) long-context vision-language models (VLMs). Our contributions are twofold: First, we frame temporal search as a Long Video Haystack problem: finding a minimal set of relevant frames (e.g., one to five) from tens of thousands based on specific queries. Upon this formulation, we introduce LV-Haystack, the first dataset with 480 hours of videos, 15,092 human-annotated instances for both training and evaluation aiming to improve temporal search quality and efficiency. Results on LV-Haystack highlight a significant research gap in temporal search capabilities, with current SOTA search methods only achieving 2.1% temporal F1 score on the Longvideobench subset. Next, inspired by visual search in images, we propose a lightweight temporal search framework, T that reframes costly temporal search as spatial search. T* leverages powerful visual localization techniques commonly used in images and introduces an adaptive zooming-in mechanism that operates across both temporal and spatial dimensions. Extensive experiments show that integrating T* with existing methods significantly improves SOTA long-form video understanding. Under an inference budget of 32 frames, T* improves GPT-4o's performance from 50.5% to 53.1% and LLaVA-OneVision-OV-72B's performance from 56.5% to 62.4% on the Longvideobench XL subset. Our code, benchmark, and models are provided in the Supplementary material.
title	T*: Re-thinking Temporal Search for Long-Form Video Understanding
topic	Computer Vision and Pattern Recognition
url	https://arxiv.org/abs/2504.02259