Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909320227061760 |
|---|---|
| author | Ma, Lingni Ye, Yuting Hong, Fangzhou Guzov, Vladimir Jiang, Yifeng Postyeni, Rowan Pesqueira, Luis Gamino, Alexander Baiyya, Vijay Kim, Hyo Jin Bailey, Kevin Fosas, David Soriano Liu, C. Karen Liu, Ziwei Engel, Jakob De Nardi, Renzo Newcombe, Richard |
| author_facet | Ma, Lingni Ye, Yuting Hong, Fangzhou Guzov, Vladimir Jiang, Yifeng Postyeni, Rowan Pesqueira, Luis Gamino, Alexander Baiyya, Vijay Kim, Hyo Jin Bailey, Kevin Fosas, David Soriano Liu, C. Karen Liu, Ziwei Engel, Jakob De Nardi, Renzo Newcombe, Richard |
| contents | We introduce Nymeria - a large-scale, diverse, richly annotated human motion dataset collected in the wild with multiple multimodal egocentric devices. The dataset comes with a) full-body ground-truth motion; b) multiple multimodal egocentric data from Project Aria devices with videos, eye tracking, IMUs and etc; and c) a third-person perspective by an additional observer. All devices are precisely synchronized and localized in on metric 3D world. We derive hierarchical protocol to add in-context language descriptions of human motion, from fine-grain motion narration, to simplified atomic action and high-level activity summarization. To the best of our knowledge, Nymeria dataset is the world's largest collection of human motion in the wild; first of its kind to provide synchronized and localized multi-device multimodal egocentric data; and the world's largest motion-language dataset. It provides 300 hours of daily activities from 264 participants across 50 locations, total travelling distance over 399Km. The language descriptions contain 301.5K sentences in 8.64M words from a vocabulary size of 6545. To demonstrate the potential of the dataset, we evaluate several SOTA algorithms for egocentric body tracking, motion synthesis, and action recognition. Data and code are open-sourced for research (c.f. https://www.projectaria.com/datasets/nymeria). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_09905 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild Ma, Lingni Ye, Yuting Hong, Fangzhou Guzov, Vladimir Jiang, Yifeng Postyeni, Rowan Pesqueira, Luis Gamino, Alexander Baiyya, Vijay Kim, Hyo Jin Bailey, Kevin Fosas, David Soriano Liu, C. Karen Liu, Ziwei Engel, Jakob De Nardi, Renzo Newcombe, Richard Computer Vision and Pattern Recognition Graphics We introduce Nymeria - a large-scale, diverse, richly annotated human motion dataset collected in the wild with multiple multimodal egocentric devices. The dataset comes with a) full-body ground-truth motion; b) multiple multimodal egocentric data from Project Aria devices with videos, eye tracking, IMUs and etc; and c) a third-person perspective by an additional observer. All devices are precisely synchronized and localized in on metric 3D world. We derive hierarchical protocol to add in-context language descriptions of human motion, from fine-grain motion narration, to simplified atomic action and high-level activity summarization. To the best of our knowledge, Nymeria dataset is the world's largest collection of human motion in the wild; first of its kind to provide synchronized and localized multi-device multimodal egocentric data; and the world's largest motion-language dataset. It provides 300 hours of daily activities from 264 participants across 50 locations, total travelling distance over 399Km. The language descriptions contain 301.5K sentences in 8.64M words from a vocabulary size of 6545. To demonstrate the potential of the dataset, we evaluate several SOTA algorithms for egocentric body tracking, motion synthesis, and action recognition. Data and code are open-sourced for research (c.f. https://www.projectaria.com/datasets/nymeria). |
| title | Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild |
| topic | Computer Vision and Pattern Recognition Graphics |
| url | https://arxiv.org/abs/2406.09905 |