Temporal Gains, Spatial Costs: Revisiting Video Fine-Tuning in Multimodal Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911525572182016 |
|---|---|
| author | Zhang, Linghao Li, Jungang Hei, Yonghua Tao, Sicheng Dai, Song Yan, Yibo Dongfang, Zihao Liu, Weiting Qin, Chenxi Li, Hanqian Zou, Xin Zhang, Jiahao Xun, Shuhang Jiang, Haiyun Hu, Xuming |
| author_facet | Zhang, Linghao Li, Jungang Hei, Yonghua Tao, Sicheng Dai, Song Yan, Yibo Dongfang, Zihao Liu, Weiting Qin, Chenxi Li, Hanqian Zou, Xin Zhang, Jiahao Xun, Shuhang Jiang, Haiyun Hu, Xuming |
| contents | Multimodal large language models (MLLMs) are typically trained in multiple stages, with video-based supervised fine-tuning (Video-SFT) serving as a key step for improving visual understanding. Yet its effect on the fine-grained evolution of visual capabilities, particularly the balance between spatial and temporal understanding, remains poorly understood. In this paper, we systematically study how Video-SFT reshapes visual capabilities in MLLMs. Across architectures, parameter scales, and frame sampling settings, we observe a consistent pattern: Video-SFT reliably improves video performance, but often yields limited gains or even degradation on static image benchmarks. We further show that this trade-off is closely tied to temporal budget: increasing the number of sampled frames generally improves video performance, but does not reliably improve static image performance. Motivated by this finding, we study an instruction-aware Hybrid-Frame strategy that adaptively allocates frame counts and partially mitigates the image-video trade-off. Our results indicate that Video-SFT is not a free lunch for MLLMs, and preserving spatial understanding remains a central challenge in joint image-video training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_17541 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Temporal Gains, Spatial Costs: Revisiting Video Fine-Tuning in Multimodal Large Language Models Zhang, Linghao Li, Jungang Hei, Yonghua Tao, Sicheng Dai, Song Yan, Yibo Dongfang, Zihao Liu, Weiting Qin, Chenxi Li, Hanqian Zou, Xin Zhang, Jiahao Xun, Shuhang Jiang, Haiyun Hu, Xuming Computer Vision and Pattern Recognition Multimodal large language models (MLLMs) are typically trained in multiple stages, with video-based supervised fine-tuning (Video-SFT) serving as a key step for improving visual understanding. Yet its effect on the fine-grained evolution of visual capabilities, particularly the balance between spatial and temporal understanding, remains poorly understood. In this paper, we systematically study how Video-SFT reshapes visual capabilities in MLLMs. Across architectures, parameter scales, and frame sampling settings, we observe a consistent pattern: Video-SFT reliably improves video performance, but often yields limited gains or even degradation on static image benchmarks. We further show that this trade-off is closely tied to temporal budget: increasing the number of sampled frames generally improves video performance, but does not reliably improve static image performance. Motivated by this finding, we study an instruction-aware Hybrid-Frame strategy that adaptively allocates frame counts and partially mitigates the image-video trade-off. Our results indicate that Video-SFT is not a free lunch for MLLMs, and preserving spatial understanding remains a central challenge in joint image-video training. |
| title | Temporal Gains, Spatial Costs: Revisiting Video Fine-Tuning in Multimodal Large Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.17541 |