Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915432494006272 |
|---|---|
| author | Xu, Liang Yang, Chengqun Lin, Zili Xu, Fei Liu, Yifan Xu, Congsheng Zhang, Yiyi Qin, Jie Sheng, Xingdong Liu, Yunhui Jin, Xin Yan, Yichao Zeng, Wenjun Yang, Xiaokang |
| author_facet | Xu, Liang Yang, Chengqun Lin, Zili Xu, Fei Liu, Yifan Xu, Congsheng Zhang, Yiyi Qin, Jie Sheng, Xingdong Liu, Yunhui Jin, Xin Yan, Yichao Zeng, Wenjun Yang, Xiaokang |
| contents | Learning action models from real-world human-centric interaction datasets is important towards building general-purpose intelligent assistants with efficiency. However, most existing datasets only offer specialist interaction category and ignore that AI assistants perceive and act based on first-person acquisition. We urge that both the generalist interaction knowledge and egocentric modality are indispensable. In this paper, we embed the manual-assisted task into a vision-language-action framework, where the assistant provides services to the instructor following egocentric vision and commands. With our hybrid RGB-MoCap system, pairs of assistants and instructors engage with multiple objects and the scene following GPT-generated scripts. Under this setting, we accomplish InterVLA, the first large-scale human-object-human interaction dataset with 11.4 hours and 1.2M frames of multimodal data, spanning 2 egocentric and 5 exocentric videos, accurate human/object motions and verbal commands. Furthermore, we establish novel benchmarks on egocentric human motion estimation, interaction synthesis, and interaction prediction with comprehensive analysis. We believe that our InterVLA testbed and the benchmarks will foster future works on building AI agents in the physical world. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_04681 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions Xu, Liang Yang, Chengqun Lin, Zili Xu, Fei Liu, Yifan Xu, Congsheng Zhang, Yiyi Qin, Jie Sheng, Xingdong Liu, Yunhui Jin, Xin Yan, Yichao Zeng, Wenjun Yang, Xiaokang Computer Vision and Pattern Recognition Learning action models from real-world human-centric interaction datasets is important towards building general-purpose intelligent assistants with efficiency. However, most existing datasets only offer specialist interaction category and ignore that AI assistants perceive and act based on first-person acquisition. We urge that both the generalist interaction knowledge and egocentric modality are indispensable. In this paper, we embed the manual-assisted task into a vision-language-action framework, where the assistant provides services to the instructor following egocentric vision and commands. With our hybrid RGB-MoCap system, pairs of assistants and instructors engage with multiple objects and the scene following GPT-generated scripts. Under this setting, we accomplish InterVLA, the first large-scale human-object-human interaction dataset with 11.4 hours and 1.2M frames of multimodal data, spanning 2 egocentric and 5 exocentric videos, accurate human/object motions and verbal commands. Furthermore, we establish novel benchmarks on egocentric human motion estimation, interaction synthesis, and interaction prediction with comprehensive analysis. We believe that our InterVLA testbed and the benchmarks will foster future works on building AI agents in the physical world. |
| title | Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2508.04681 |