Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929552239886336 |
|---|---|
| author | Yang, Gengshan Bajcsy, Andrea Saito, Shunsuke Kanazawa, Angjoo |
| author_facet | Yang, Gengshan Bajcsy, Andrea Saito, Shunsuke Kanazawa, Angjoo |
| contents | We present Agent-to-Sim (ATS), a framework for learning interactive behavior models of 3D agents from casual longitudinal video collections. Different from prior works that rely on marker-based tracking and multiview cameras, ATS learns natural behaviors of animal and human agents non-invasively through video observations recorded over a long time-span (e.g., a month) in a single environment. Modeling 3D behavior of an agent requires persistent 3D tracking (e.g., knowing which point corresponds to which) over a long time period. To obtain such data, we develop a coarse-to-fine registration method that tracks the agent and the camera over time through a canonical 3D space, resulting in a complete and persistent spacetime 4D representation. We then train a generative model of agent behaviors using paired data of perception and motion of an agent queried from the 4D reconstruction. ATS enables real-to-sim transfer from video recordings of an agent to an interactive behavior simulator. We demonstrate results on pets (e.g., cat, dog, bunny) and human given monocular RGBD videos captured by a smartphone. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_16259 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos Yang, Gengshan Bajcsy, Andrea Saito, Shunsuke Kanazawa, Angjoo Computer Vision and Pattern Recognition Graphics Robotics We present Agent-to-Sim (ATS), a framework for learning interactive behavior models of 3D agents from casual longitudinal video collections. Different from prior works that rely on marker-based tracking and multiview cameras, ATS learns natural behaviors of animal and human agents non-invasively through video observations recorded over a long time-span (e.g., a month) in a single environment. Modeling 3D behavior of an agent requires persistent 3D tracking (e.g., knowing which point corresponds to which) over a long time period. To obtain such data, we develop a coarse-to-fine registration method that tracks the agent and the camera over time through a canonical 3D space, resulting in a complete and persistent spacetime 4D representation. We then train a generative model of agent behaviors using paired data of perception and motion of an agent queried from the 4D reconstruction. ATS enables real-to-sim transfer from video recordings of an agent to an interactive behavior simulator. We demonstrate results on pets (e.g., cat, dog, bunny) and human given monocular RGBD videos captured by a smartphone. |
| title | Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos |
| topic | Computer Vision and Pattern Recognition Graphics Robotics |
| url | https://arxiv.org/abs/2410.16259 |