Ego-InBetween: Generating Object State Transitions in Ego-Centric Videos
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914489314574336 |
|---|---|
| author | Ge, Mengmeng Isobe, Takashi Jia, Xu Sun, Yanan Yang, Zetong Wang, Weinong Zhou, Dong Li, Dong Lu, Huchuan Barsoum, Emad |
| author_facet | Ge, Mengmeng Isobe, Takashi Jia, Xu Sun, Yanan Yang, Zetong Wang, Weinong Zhou, Dong Li, Dong Lu, Huchuan Barsoum, Emad |
| contents | Understanding physical transformation processes is crucial for both human cognition and artificial intelligence systems, particularly from an egocentric perspective, which serves as a key bridge between humans and machines in action modeling. We define this modeling process as Egocentric Instructed Visual State Transition (EIVST), which involves generating intermediate frames that depict object transformations between initial and target states under a brief action instruction. EIVST poses two challenges for current generative models: (1) understanding the visual scenes of the initial and target states and reasoning about transformation steps from an egocentric view, and (2) generating a consistent intermediate transition that follows the given instruction while preserving object appearance across the two visual states. To address these challenges, we propose the EgoIn framework. It first infers the multi-step transition process between two given states using TransitionVLM, fine-tuned on our curated dataset to better adapt to this task and reduce hallucinated information. It then generates a sequence of frames based on transition conditions produced by the proposed Transition Conditioning module. Additionally, we introduce Object-aware Auxiliary Supervision to preserve consistent object appearance throughout the transition. Extensive experiments on human-object and robot-object interaction datasets demonstrate EgoIn's superior performance in generating semantically meaningful and visually coherent transformation sequences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_17749 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Ego-InBetween: Generating Object State Transitions in Ego-Centric Videos Ge, Mengmeng Isobe, Takashi Jia, Xu Sun, Yanan Yang, Zetong Wang, Weinong Zhou, Dong Li, Dong Lu, Huchuan Barsoum, Emad Computer Vision and Pattern Recognition Understanding physical transformation processes is crucial for both human cognition and artificial intelligence systems, particularly from an egocentric perspective, which serves as a key bridge between humans and machines in action modeling. We define this modeling process as Egocentric Instructed Visual State Transition (EIVST), which involves generating intermediate frames that depict object transformations between initial and target states under a brief action instruction. EIVST poses two challenges for current generative models: (1) understanding the visual scenes of the initial and target states and reasoning about transformation steps from an egocentric view, and (2) generating a consistent intermediate transition that follows the given instruction while preserving object appearance across the two visual states. To address these challenges, we propose the EgoIn framework. It first infers the multi-step transition process between two given states using TransitionVLM, fine-tuned on our curated dataset to better adapt to this task and reduce hallucinated information. It then generates a sequence of frames based on transition conditions produced by the proposed Transition Conditioning module. Additionally, we introduce Object-aware Auxiliary Supervision to preserve consistent object appearance throughout the transition. Extensive experiments on human-object and robot-object interaction datasets demonstrate EgoIn's superior performance in generating semantically meaningful and visually coherent transformation sequences. |
| title | Ego-InBetween: Generating Object State Transitions in Ego-Centric Videos |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2604.17749 |