Image de couverture de livre

Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Grauman, Kristen, Westbury, Andrew, Torresani, Lorenzo, Kitani, Kris, Malik, Jitendra, Afouras, Triantafyllos, Ashutosh, Kumar, Baiyya, Vijay, Bansal, Siddhant, Boote, Bikram, Byrne, Eugene, Chavis, Zach, Chen, Joya, Cheng, Feng, Chu, Fu-Jen, Crane, Sean, Dasgupta, Avijit, Dong, Jing, Escobar, Maria, Forigua, Cristhian, Gebreselasie, Abrham, Haresh, Sanjay, Huang, Jing, Islam, Md Mohaiminul, Jain, Suyog, Khirodkar, Rawal, Kukreja, Devansh, Liang, Kevin J, Liu, Jia-Wei, Majumder, Sagnik, Mao, Yongsen, Martin, Miguel, Mavroudi, Effrosyni, Nagarajan, Tushar, Ragusa, Francesco, Ramakrishnan, Santhosh Kumar, Seminara, Luigi, Somayazulu, Arjun, Song, Yale, Su, Shan, Xue, Zihui, Zhang, Edward, Zhang, Jinxu, Castillo, Angela, Chen, Changan, Fu, Xinzhu, Furuta, Ryosuke, Gonzalez, Cristina, Gupta, Prince, Hu, Jiabo, Huang, Yifei, Huang, Yiming, Khoo, Weslie, Kumar, Anush, Kuo, Robert, Lakhavani, Sach, Liu, Miao, Luo, Mi, Luo, Zhengyi, Meredith, Brighid, Miller, Austin, Oguntola, Oluwatumininu, Pan, Xiaqing, Peng, Penny, Pramanick, Shraman, Ramazanova, Merey, Ryan, Fiona, Shan, Wei, Somasundaram, Kiran, Song, Chenan, Southerland, Audrey, Tateno, Masatoshi, Wang, Huiyu, Wang, Yuchen, Yagi, Takuma, Yan, Mingfei, Yang, Xitong, Yu, Zecheng, Zha, Shengxin Cindy, Zhao, Chen, Zhao, Ziwei, Zhu, Zhifan, Zhuo, Jeff, Arbelaez, Pablo, Bertasius, Gedas, Crandall, David, Damen, Dima, Engel, Jakob, Farinella, Giovanni Maria, Furnari, Antonino, Ghanem, Bernard, Hoffman, Judy, Jawahar, C. V., Newcombe, Richard, Park, Hyun Soo, Rehg, James M., Sato, Yoichi, Savva, Manolis, Shi, Jianbo, Shou, Mike Zheng, Wray, Michael
Format:	Preprint
Publié:	2023
Sujets:	Computer Vision and Pattern Recognition Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

_version_	1866916411258961920
author	Grauman, Kristen Westbury, Andrew Torresani, Lorenzo Kitani, Kris Malik, Jitendra Afouras, Triantafyllos Ashutosh, Kumar Baiyya, Vijay Bansal, Siddhant Boote, Bikram Byrne, Eugene Chavis, Zach Chen, Joya Cheng, Feng Chu, Fu-Jen Crane, Sean Dasgupta, Avijit Dong, Jing Escobar, Maria Forigua, Cristhian Gebreselasie, Abrham Haresh, Sanjay Huang, Jing Islam, Md Mohaiminul Jain, Suyog Khirodkar, Rawal Kukreja, Devansh Liang, Kevin J Liu, Jia-Wei Majumder, Sagnik Mao, Yongsen Martin, Miguel Mavroudi, Effrosyni Nagarajan, Tushar Ragusa, Francesco Ramakrishnan, Santhosh Kumar Seminara, Luigi Somayazulu, Arjun Song, Yale Su, Shan Xue, Zihui Zhang, Edward Zhang, Jinxu Castillo, Angela Chen, Changan Fu, Xinzhu Furuta, Ryosuke Gonzalez, Cristina Gupta, Prince Hu, Jiabo Huang, Yifei Huang, Yiming Khoo, Weslie Kumar, Anush Kuo, Robert Lakhavani, Sach Liu, Miao Luo, Mi Luo, Zhengyi Meredith, Brighid Miller, Austin Oguntola, Oluwatumininu Pan, Xiaqing Peng, Penny Pramanick, Shraman Ramazanova, Merey Ryan, Fiona Shan, Wei Somasundaram, Kiran Song, Chenan Southerland, Audrey Tateno, Masatoshi Wang, Huiyu Wang, Yuchen Yagi, Takuma Yan, Mingfei Yang, Xitong Yu, Zecheng Zha, Shengxin Cindy Zhao, Chen Zhao, Ziwei Zhu, Zhifan Zhuo, Jeff Arbelaez, Pablo Bertasius, Gedas Crandall, David Damen, Dima Engel, Jakob Farinella, Giovanni Maria Furnari, Antonino Ghanem, Bernard Hoffman, Judy Jawahar, C. V. Newcombe, Richard Park, Hyun Soo Rehg, James M. Sato, Yoichi Savva, Manolis Shi, Jianbo Shou, Mike Zheng Wray, Michael
author_facet	Grauman, Kristen Westbury, Andrew Torresani, Lorenzo Kitani, Kris Malik, Jitendra Afouras, Triantafyllos Ashutosh, Kumar Baiyya, Vijay Bansal, Siddhant Boote, Bikram Byrne, Eugene Chavis, Zach Chen, Joya Cheng, Feng Chu, Fu-Jen Crane, Sean Dasgupta, Avijit Dong, Jing Escobar, Maria Forigua, Cristhian Gebreselasie, Abrham Haresh, Sanjay Huang, Jing Islam, Md Mohaiminul Jain, Suyog Khirodkar, Rawal Kukreja, Devansh Liang, Kevin J Liu, Jia-Wei Majumder, Sagnik Mao, Yongsen Martin, Miguel Mavroudi, Effrosyni Nagarajan, Tushar Ragusa, Francesco Ramakrishnan, Santhosh Kumar Seminara, Luigi Somayazulu, Arjun Song, Yale Su, Shan Xue, Zihui Zhang, Edward Zhang, Jinxu Castillo, Angela Chen, Changan Fu, Xinzhu Furuta, Ryosuke Gonzalez, Cristina Gupta, Prince Hu, Jiabo Huang, Yifei Huang, Yiming Khoo, Weslie Kumar, Anush Kuo, Robert Lakhavani, Sach Liu, Miao Luo, Mi Luo, Zhengyi Meredith, Brighid Miller, Austin Oguntola, Oluwatumininu Pan, Xiaqing Peng, Penny Pramanick, Shraman Ramazanova, Merey Ryan, Fiona Shan, Wei Somasundaram, Kiran Song, Chenan Southerland, Audrey Tateno, Masatoshi Wang, Huiyu Wang, Yuchen Yagi, Takuma Yan, Mingfei Yang, Xitong Yu, Zecheng Zha, Shengxin Cindy Zhao, Chen Zhao, Ziwei Zhu, Zhifan Zhuo, Jeff Arbelaez, Pablo Bertasius, Gedas Crandall, David Damen, Dima Engel, Jakob Farinella, Giovanni Maria Furnari, Antonino Ghanem, Bernard Hoffman, Judy Jawahar, C. V. Newcombe, Richard Park, Hyun Soo Rehg, James M. Sato, Yoichi Savva, Manolis Shi, Jianbo Shou, Mike Zheng Wray, Michael
contents	We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge. Ego-Exo4D centers around simultaneously-captured egocentric and exocentric video of skilled human activities (e.g., sports, music, dance, bike repair). 740 participants from 13 cities worldwide performed these activities in 123 different natural scene contexts, yielding long-form captures from 1 to 42 minutes each and 1,286 hours of video combined. The multimodal nature of the dataset is unprecedented: the video is accompanied by multichannel audio, eye gaze, 3D point clouds, camera poses, IMU, and multiple paired language descriptions -- including a novel "expert commentary" done by coaches and teachers and tailored to the skilled-activity domain. To push the frontier of first-person video understanding of skilled human activity, we also present a suite of benchmark tasks and their annotations, including fine-grained activity understanding, proficiency estimation, cross-view translation, and 3D hand/body pose. All resources are open sourced to fuel new research in the community. Project page: http://ego-exo4d-data.org/
format	Preprint
id	arxiv_https___arxiv_org_abs_2311_18259
institution	arXiv
publishDate	2023
record_format	arxiv
spellingShingle	Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives Grauman, Kristen Westbury, Andrew Torresani, Lorenzo Kitani, Kris Malik, Jitendra Afouras, Triantafyllos Ashutosh, Kumar Baiyya, Vijay Bansal, Siddhant Boote, Bikram Byrne, Eugene Chavis, Zach Chen, Joya Cheng, Feng Chu, Fu-Jen Crane, Sean Dasgupta, Avijit Dong, Jing Escobar, Maria Forigua, Cristhian Gebreselasie, Abrham Haresh, Sanjay Huang, Jing Islam, Md Mohaiminul Jain, Suyog Khirodkar, Rawal Kukreja, Devansh Liang, Kevin J Liu, Jia-Wei Majumder, Sagnik Mao, Yongsen Martin, Miguel Mavroudi, Effrosyni Nagarajan, Tushar Ragusa, Francesco Ramakrishnan, Santhosh Kumar Seminara, Luigi Somayazulu, Arjun Song, Yale Su, Shan Xue, Zihui Zhang, Edward Zhang, Jinxu Castillo, Angela Chen, Changan Fu, Xinzhu Furuta, Ryosuke Gonzalez, Cristina Gupta, Prince Hu, Jiabo Huang, Yifei Huang, Yiming Khoo, Weslie Kumar, Anush Kuo, Robert Lakhavani, Sach Liu, Miao Luo, Mi Luo, Zhengyi Meredith, Brighid Miller, Austin Oguntola, Oluwatumininu Pan, Xiaqing Peng, Penny Pramanick, Shraman Ramazanova, Merey Ryan, Fiona Shan, Wei Somasundaram, Kiran Song, Chenan Southerland, Audrey Tateno, Masatoshi Wang, Huiyu Wang, Yuchen Yagi, Takuma Yan, Mingfei Yang, Xitong Yu, Zecheng Zha, Shengxin Cindy Zhao, Chen Zhao, Ziwei Zhu, Zhifan Zhuo, Jeff Arbelaez, Pablo Bertasius, Gedas Crandall, David Damen, Dima Engel, Jakob Farinella, Giovanni Maria Furnari, Antonino Ghanem, Bernard Hoffman, Judy Jawahar, C. V. Newcombe, Richard Park, Hyun Soo Rehg, James M. Sato, Yoichi Savva, Manolis Shi, Jianbo Shou, Mike Zheng Wray, Michael Computer Vision and Pattern Recognition Artificial Intelligence We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge. Ego-Exo4D centers around simultaneously-captured egocentric and exocentric video of skilled human activities (e.g., sports, music, dance, bike repair). 740 participants from 13 cities worldwide performed these activities in 123 different natural scene contexts, yielding long-form captures from 1 to 42 minutes each and 1,286 hours of video combined. The multimodal nature of the dataset is unprecedented: the video is accompanied by multichannel audio, eye gaze, 3D point clouds, camera poses, IMU, and multiple paired language descriptions -- including a novel "expert commentary" done by coaches and teachers and tailored to the skilled-activity domain. To push the frontier of first-person video understanding of skilled human activity, we also present a suite of benchmark tasks and their annotations, including fine-grained activity understanding, proficiency estimation, cross-view translation, and 3D hand/body pose. All resources are open sourced to fuel new research in the community. Project page: http://ego-exo4d-data.org/
title	Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
topic	Computer Vision and Pattern Recognition Artificial Intelligence
url	https://arxiv.org/abs/2311.18259