Understanding Physical Dynamics with Counterfactual World Modeling
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929428155596800 |
|---|---|
| author | Venkatesh, Rahul Chen, Honglin Feigelis, Kevin Bear, Daniel M. Jedoui, Khaled Kotar, Klemen Binder, Felix Lee, Wanhee Liu, Sherry Smith, Kevin A. Fan, Judith E. Yamins, Daniel L. K. |
| author_facet | Venkatesh, Rahul Chen, Honglin Feigelis, Kevin Bear, Daniel M. Jedoui, Khaled Kotar, Klemen Binder, Felix Lee, Wanhee Liu, Sherry Smith, Kevin A. Fan, Judith E. Yamins, Daniel L. K. |
| contents | The ability to understand physical dynamics is critical for agents to act in the world. Here, we use Counterfactual World Modeling (CWM) to extract vision structures for dynamics understanding. CWM uses a temporally-factored masking policy for masked prediction of video data without annotations. This policy enables highly effective "counterfactual prompting" of the predictor, allowing a spectrum of visual structures to be extracted from a single pre-trained predictor without finetuning on annotated datasets. We demonstrate that these structures are useful for physical dynamics understanding, allowing CWM to achieve the state-of-the-art performance on the Physion benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_06721 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Understanding Physical Dynamics with Counterfactual World Modeling Venkatesh, Rahul Chen, Honglin Feigelis, Kevin Bear, Daniel M. Jedoui, Khaled Kotar, Klemen Binder, Felix Lee, Wanhee Liu, Sherry Smith, Kevin A. Fan, Judith E. Yamins, Daniel L. K. Computer Vision and Pattern Recognition The ability to understand physical dynamics is critical for agents to act in the world. Here, we use Counterfactual World Modeling (CWM) to extract vision structures for dynamics understanding. CWM uses a temporally-factored masking policy for masked prediction of video data without annotations. This policy enables highly effective "counterfactual prompting" of the predictor, allowing a spectrum of visual structures to be extracted from a single pre-trained predictor without finetuning on annotated datasets. We demonstrate that these structures are useful for physical dynamics understanding, allowing CWM to achieve the state-of-the-art performance on the Physion benchmark. |
| title | Understanding Physical Dynamics with Counterfactual World Modeling |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.06721 |