Learning Invariant Visual Representations for Planning with Joint-Embedding Predictive World Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918352647094272 |
|---|---|
| author | Toso, Leonardo F. Shadunts, Davit Lu, Yunyang Sharma, Nihal Zhan, Donglin Nguyen, Nam H. Anderson, James |
| author_facet | Toso, Leonardo F. Shadunts, Davit Lu, Yunyang Sharma, Nihal Zhan, Donglin Nguyen, Nam H. Anderson, James |
| contents | World models learned from high-dimensional visual observations allow agents to make decisions and plan directly in latent space, avoiding pixel-level reconstruction. However, recent latent predictive architectures (JEPAs), including the DINO world model (DINO-WM), display a degradation in test time robustness due to their sensitivity to "slow features". These include visual variations such as background changes and distractors that are irrelevant to the task being solved. We address this limitation by augmenting the predictive objective with a bisimulation encoder that enforces control-relevant state equivalence, mapping states with similar transition dynamics to nearby latent states while limiting contributions from slow features. We evaluate our model on a simple navigation task under different test-time background changes and visual distractors. Across all benchmarks, our model consistently improves robustness to slow features while operating in a reduced latent space, up to 10x smaller than that of DINO-WM. Moreover, our model is agnostic to the choice of pretrained visual encoder and maintains robustness when paired with DINOv2, SimDINOv2, and iBOT features. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_18639 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Learning Invariant Visual Representations for Planning with Joint-Embedding Predictive World Models Toso, Leonardo F. Shadunts, Davit Lu, Yunyang Sharma, Nihal Zhan, Donglin Nguyen, Nam H. Anderson, James Machine Learning Optimization and Control World models learned from high-dimensional visual observations allow agents to make decisions and plan directly in latent space, avoiding pixel-level reconstruction. However, recent latent predictive architectures (JEPAs), including the DINO world model (DINO-WM), display a degradation in test time robustness due to their sensitivity to "slow features". These include visual variations such as background changes and distractors that are irrelevant to the task being solved. We address this limitation by augmenting the predictive objective with a bisimulation encoder that enforces control-relevant state equivalence, mapping states with similar transition dynamics to nearby latent states while limiting contributions from slow features. We evaluate our model on a simple navigation task under different test-time background changes and visual distractors. Across all benchmarks, our model consistently improves robustness to slow features while operating in a reduced latent space, up to 10x smaller than that of DINO-WM. Moreover, our model is agnostic to the choice of pretrained visual encoder and maintains robustness when paired with DINOv2, SimDINOv2, and iBOT features. |
| title | Learning Invariant Visual Representations for Planning with Joint-Embedding Predictive World Models |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2602.18639 |