LinkedOut: Linking World Knowledge Representation Out of Video LLM for Next-Generation Video Recommendation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914208185057280 |
|---|---|
| author | Zhang, Haichao Lu, Yao Wang, Lichen Li, Yunzhe Chen, Daiwei Xu, Yunpeng Fu, Yun |
| author_facet | Zhang, Haichao Lu, Yao Wang, Lichen Li, Yunzhe Chen, Daiwei Xu, Yunpeng Fu, Yun |
| contents | Video Large Language Models (VLLMs) unlock world-knowledge-aware video understanding through pretraining on internet-scale data and have already shown promise on tasks such as movie analysis and video question answering. However, deploying VLLMs for downstream tasks such as video recommendation remains challenging, since real systems require multi-video inputs, lightweight backbones, low-latency sequential inference, and rapid response. In practice, (1) decode-only generation yields high latency for sequential inference, (2) typical interfaces do not support multi-video inputs, and (3) constraining outputs to language discards fine-grained visual details that matter for downstream vision tasks. We argue that these limitations stem from the absence of a representation that preserves pixel-level detail while leveraging world knowledge. We present LinkedOut, a representation that extracts VLLM world knowledge directly from video to enable fast inference, supports multi-video histories, and removes the language bottleneck. LinkedOut extracts semantically grounded, knowledge-aware tokens from raw frames using VLLMs, guided by promptable queries and optional auxiliary modalities. We introduce a cross-layer knowledge fusion MoE that selects the appropriate level of abstraction from the rich VLLM features, enabling personalized, interpretable, and low-latency recommendation. To our knowledge, LinkedOut is the first VLLM-based video recommendation method that operates on raw frames without handcrafted labels, achieving state-of-the-art results on standard benchmarks. Interpretability studies and ablations confirm the benefits of layer diversity and layer-wise fusion, pointing to a practical path that fully leverages VLLM world-knowledge priors and visual reasoning for downstream vision tasks such as recommendation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_16891 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LinkedOut: Linking World Knowledge Representation Out of Video LLM for Next-Generation Video Recommendation Zhang, Haichao Lu, Yao Wang, Lichen Li, Yunzhe Chen, Daiwei Xu, Yunpeng Fu, Yun Computer Vision and Pattern Recognition Artificial Intelligence Information Retrieval Machine Learning Multimedia 68T05, 68T07, 68T10, 68T45, 68T50, 68U10, 68P20, 62H30, 62H35 I.2.4; I.2.6; I.2.7; I.2.8; I.2.10; I.4; I.5; I.7; H.3.1; H.3.3; H.3.4; H.3.5 Video Large Language Models (VLLMs) unlock world-knowledge-aware video understanding through pretraining on internet-scale data and have already shown promise on tasks such as movie analysis and video question answering. However, deploying VLLMs for downstream tasks such as video recommendation remains challenging, since real systems require multi-video inputs, lightweight backbones, low-latency sequential inference, and rapid response. In practice, (1) decode-only generation yields high latency for sequential inference, (2) typical interfaces do not support multi-video inputs, and (3) constraining outputs to language discards fine-grained visual details that matter for downstream vision tasks. We argue that these limitations stem from the absence of a representation that preserves pixel-level detail while leveraging world knowledge. We present LinkedOut, a representation that extracts VLLM world knowledge directly from video to enable fast inference, supports multi-video histories, and removes the language bottleneck. LinkedOut extracts semantically grounded, knowledge-aware tokens from raw frames using VLLMs, guided by promptable queries and optional auxiliary modalities. We introduce a cross-layer knowledge fusion MoE that selects the appropriate level of abstraction from the rich VLLM features, enabling personalized, interpretable, and low-latency recommendation. To our knowledge, LinkedOut is the first VLLM-based video recommendation method that operates on raw frames without handcrafted labels, achieving state-of-the-art results on standard benchmarks. Interpretability studies and ablations confirm the benefits of layer diversity and layer-wise fusion, pointing to a practical path that fully leverages VLLM world-knowledge priors and visual reasoning for downstream vision tasks such as recommendation. |
| title | LinkedOut: Linking World Knowledge Representation Out of Video LLM for Next-Generation Video Recommendation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Information Retrieval Machine Learning Multimedia 68T05, 68T07, 68T10, 68T45, 68T50, 68U10, 68P20, 62H30, 62H35 I.2.4; I.2.6; I.2.7; I.2.8; I.2.10; I.4; I.5; I.7; H.3.1; H.3.3; H.3.4; H.3.5 |
| url | https://arxiv.org/abs/2512.16891 |