Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.06939 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909248517046272 |
|---|---|
| author | Yenamandra, Sriram Ramachandran, Arun Khanna, Mukul Yadav, Karmesh Vakil, Jay Melnik, Andrew Büttner, Michael Harz, Leon Brown, Lyon Nandi, Gora Chand PS, Arjun Yadav, Gaurav Kumar Kala, Rahul Haschke, Robert Luo, Yang Zhu, Jinxin Han, Yansen Lu, Bingyi Gu, Xuan Liu, Qinyuan Zhao, Yaping Ye, Qiting Dou, Chenxiao Chua, Yansong Kuzma, Volodymyr Humennyy, Vladyslav Partsey, Ruslan Francis, Jonathan Chaplot, Devendra Singh Chhablani, Gunjan Clegg, Alexander Gervet, Theophile Jain, Vidhi Ramrakhya, Ram Szot, Andrew Wang, Austin Yang, Tsung-Yen Edsinger, Aaron Kemp, Charlie Shah, Binit Kira, Zsolt Batra, Dhruv Mottaghi, Roozbeh Bisk, Yonatan Paxton, Chris |
| author_facet | Yenamandra, Sriram Ramachandran, Arun Khanna, Mukul Yadav, Karmesh Vakil, Jay Melnik, Andrew Büttner, Michael Harz, Leon Brown, Lyon Nandi, Gora Chand PS, Arjun Yadav, Gaurav Kumar Kala, Rahul Haschke, Robert Luo, Yang Zhu, Jinxin Han, Yansen Lu, Bingyi Gu, Xuan Liu, Qinyuan Zhao, Yaping Ye, Qiting Dou, Chenxiao Chua, Yansong Kuzma, Volodymyr Humennyy, Vladyslav Partsey, Ruslan Francis, Jonathan Chaplot, Devendra Singh Chhablani, Gunjan Clegg, Alexander Gervet, Theophile Jain, Vidhi Ramrakhya, Ram Szot, Andrew Wang, Austin Yang, Tsung-Yen Edsinger, Aaron Kemp, Charlie Shah, Binit Kira, Zsolt Batra, Dhruv Mottaghi, Roozbeh Bisk, Yonatan Paxton, Chris |
| contents | In order to develop robots that can effectively serve as versatile and capable home assistants, it is crucial for them to reliably perceive and interact with a wide variety of objects across diverse environments. To this end, we proposed Open Vocabulary Mobile Manipulation as a key benchmark task for robotics: finding any object in a novel environment and placing it on any receptacle surface within that environment. We organized a NeurIPS 2023 competition featuring both simulation and real-world components to evaluate solutions to this task. Our baselines on the most challenging version of this task, using real perception in simulation, achieved only an 0.8% success rate; by the end of the competition, the best participants achieved an 10.8\% success rate, a 13x improvement. We observed that the most successful teams employed a variety of methods, yet two common threads emerged among the best solutions: enhancing error detection and recovery, and improving the integration of perception with decision-making processes. In this paper, we detail the results and methodologies used, both in simulation and real-world settings. We discuss the lessons learned and their implications for future research. Additionally, we compare performance in real and simulated environments, emphasizing the necessity for robust generalization to novel settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_06939 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challenge Yenamandra, Sriram Ramachandran, Arun Khanna, Mukul Yadav, Karmesh Vakil, Jay Melnik, Andrew Büttner, Michael Harz, Leon Brown, Lyon Nandi, Gora Chand PS, Arjun Yadav, Gaurav Kumar Kala, Rahul Haschke, Robert Luo, Yang Zhu, Jinxin Han, Yansen Lu, Bingyi Gu, Xuan Liu, Qinyuan Zhao, Yaping Ye, Qiting Dou, Chenxiao Chua, Yansong Kuzma, Volodymyr Humennyy, Vladyslav Partsey, Ruslan Francis, Jonathan Chaplot, Devendra Singh Chhablani, Gunjan Clegg, Alexander Gervet, Theophile Jain, Vidhi Ramrakhya, Ram Szot, Andrew Wang, Austin Yang, Tsung-Yen Edsinger, Aaron Kemp, Charlie Shah, Binit Kira, Zsolt Batra, Dhruv Mottaghi, Roozbeh Bisk, Yonatan Paxton, Chris Robotics Computer Vision and Pattern Recognition In order to develop robots that can effectively serve as versatile and capable home assistants, it is crucial for them to reliably perceive and interact with a wide variety of objects across diverse environments. To this end, we proposed Open Vocabulary Mobile Manipulation as a key benchmark task for robotics: finding any object in a novel environment and placing it on any receptacle surface within that environment. We organized a NeurIPS 2023 competition featuring both simulation and real-world components to evaluate solutions to this task. Our baselines on the most challenging version of this task, using real perception in simulation, achieved only an 0.8% success rate; by the end of the competition, the best participants achieved an 10.8\% success rate, a 13x improvement. We observed that the most successful teams employed a variety of methods, yet two common threads emerged among the best solutions: enhancing error detection and recovery, and improving the integration of perception with decision-making processes. In this paper, we detail the results and methodologies used, both in simulation and real-world settings. We discuss the lessons learned and their implications for future research. Additionally, we compare performance in real and simulated environments, emphasizing the necessity for robust generalization to novel settings. |
| title | Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challenge |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.06939 |