ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911538421432320 |
|---|---|
| author | Zhang, Haichao Li, Yijiang He, Shwai Nagarajan, Tushar Chen, Mingfei Lu, Jianglin Li, Ang Fu, Yun |
| author_facet | Zhang, Haichao Li, Yijiang He, Shwai Nagarajan, Tushar Chen, Mingfei Lu, Jianglin Li, Ang Fu, Yun |
| contents | Recent progress in latent world models (e.g., V-JEPA2) has shown promising capability in forecasting future world states from video observations. Nevertheless, dense prediction from a short observation window limits temporal context and can bias predictors toward local, low-level extrapolation, making it difficult to capture long-horizon semantics and reducing downstream utility. Vision--language models (VLMs), in contrast, provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames, but they are not ideal as standalone dense predictors due to compute-driven sparse sampling, a language-output bottleneck that compresses fine-grained interaction states into text-oriented representations, and a data-regime mismatch when adapting to small action-conditioned datasets. We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance via a dual-temporal pathway: a dense JEPA branch for fine-grained motion and interaction cues, and a uniformly sampled VLM \emph{thinker} branch with a larger temporal stride for knowledge-rich guidance. To transfer the VLM's progressive reasoning signals effectively, we introduce a hierarchical pyramid representation extraction module that aggregates multi-layer VLM representations into guidance features compatible with latent prediction. Experiments on hand-manipulation trajectory prediction show that our method outperforms both a strong VLM-only baseline and a JEPA-predictor baseline, and yields more robust long-horizon rollout behavior. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_22281 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model Zhang, Haichao Li, Yijiang He, Shwai Nagarajan, Tushar Chen, Mingfei Lu, Jianglin Li, Ang Fu, Yun Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Robotics 68T45, 68T07, 68U10, 68T10, 93C85, 93B40, 70Q05 I.2.9; I.2.10; I.2.6; I.2.7; I.4.8; I.4.9; I.2.8 Recent progress in latent world models (e.g., V-JEPA2) has shown promising capability in forecasting future world states from video observations. Nevertheless, dense prediction from a short observation window limits temporal context and can bias predictors toward local, low-level extrapolation, making it difficult to capture long-horizon semantics and reducing downstream utility. Vision--language models (VLMs), in contrast, provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames, but they are not ideal as standalone dense predictors due to compute-driven sparse sampling, a language-output bottleneck that compresses fine-grained interaction states into text-oriented representations, and a data-regime mismatch when adapting to small action-conditioned datasets. We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance via a dual-temporal pathway: a dense JEPA branch for fine-grained motion and interaction cues, and a uniformly sampled VLM \emph{thinker} branch with a larger temporal stride for knowledge-rich guidance. To transfer the VLM's progressive reasoning signals effectively, we introduce a hierarchical pyramid representation extraction module that aggregates multi-layer VLM representations into guidance features compatible with latent prediction. Experiments on hand-manipulation trajectory prediction show that our method outperforms both a strong VLM-only baseline and a JEPA-predictor baseline, and yields more robust long-horizon rollout behavior. |
| title | ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Robotics 68T45, 68T07, 68U10, 68T10, 93C85, 93B40, 70Q05 I.2.9; I.2.10; I.2.6; I.2.7; I.4.8; I.4.9; I.2.8 |
| url | https://arxiv.org/abs/2603.22281 |