Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.20576 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914581696217088 |
|---|---|
| author | Kao, Chia-Hsiang Huynh, Cong Phuoc Wang, Chien-Yi Vesdapunt, Noranart Stojanov, Stefan Hariharan, Bharath Obiednikov, Oleksandr Zhou, Ning |
| author_facet | Kao, Chia-Hsiang Huynh, Cong Phuoc Wang, Chien-Yi Vesdapunt, Noranart Stojanov, Stefan Hariharan, Bharath Obiednikov, Oleksandr Zhou, Ning |
| contents | Inferring rigid-body physical states and properties from monocular videos is a fundamental step toward physics-based perception and simulation. Existing approaches assume specific underlying physical systems, object types, and camera poses, making them unable to generalize to complex real-world settings. We introduce $Δ$YNAMICS, a vision-language framework that uses language as a unified representation of rigid-body dynamics. Instead of directly predicting parameters, $Δ$YNAMICS generates scene configurations in a structured text format for physics simulation. We enhance the model's generalization by integrating natural language motion reasoning and leveraging optical flow as a semantic-agnostic input. On the CLEVRER dataset, $Δ$YNAMICS achieves a segmentation IoU of 0.30, a 7x improvement over leading VLMs (InternVL3-8B, Qwen2.5-VL-7B and Claude-4-Sonnet). Additionally, test-time sampling and evolutionary search further boost performance by 27% and 120% in segmentation IoU, respectively. Finally, we demonstrate strong transfer to a new dataset of 235 real-world rigid-body videos, highlighting the potential of language-driven physics inference for bridging perception and simulation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_20576 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | $Δ$ynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videos Kao, Chia-Hsiang Huynh, Cong Phuoc Wang, Chien-Yi Vesdapunt, Noranart Stojanov, Stefan Hariharan, Bharath Obiednikov, Oleksandr Zhou, Ning Computer Vision and Pattern Recognition Inferring rigid-body physical states and properties from monocular videos is a fundamental step toward physics-based perception and simulation. Existing approaches assume specific underlying physical systems, object types, and camera poses, making them unable to generalize to complex real-world settings. We introduce $Δ$YNAMICS, a vision-language framework that uses language as a unified representation of rigid-body dynamics. Instead of directly predicting parameters, $Δ$YNAMICS generates scene configurations in a structured text format for physics simulation. We enhance the model's generalization by integrating natural language motion reasoning and leveraging optical flow as a semantic-agnostic input. On the CLEVRER dataset, $Δ$YNAMICS achieves a segmentation IoU of 0.30, a 7x improvement over leading VLMs (InternVL3-8B, Qwen2.5-VL-7B and Claude-4-Sonnet). Additionally, test-time sampling and evolutionary search further boost performance by 27% and 120% in segmentation IoU, respectively. Finally, we demonstrate strong transfer to a new dataset of 235 real-world rigid-body videos, highlighting the potential of language-driven physics inference for bridging perception and simulation. |
| title | $Δ$ynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videos |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.20576 |