RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908680459386880 |
|---|---|
| author | Atreya, Pranav Pertsch, Karl Lee, Tony Kim, Moo Jin Jain, Arhan Kuramshin, Artur Eppner, Clemens Neary, Cyrus Hu, Edward Ramos, Fabio Tremblay, Jonathan Arora, Kanav Ellis, Kirsty Macesanu, Luca Villasevil, Marcel Torne Leonard, Matthew Cho, Meedeum Aslan, Ozgur Dass, Shivin Wang, Jie Reger, William Yuan, Xingfang Yang, Xuning Gupta, Abhishek Jayaraman, Dinesh Berseth, Glen Daniilidis, Kostas Martin-Martin, Roberto Lee, Youngwoon Liang, Percy Finn, Chelsea Levine, Sergey |
| author_facet | Atreya, Pranav Pertsch, Karl Lee, Tony Kim, Moo Jin Jain, Arhan Kuramshin, Artur Eppner, Clemens Neary, Cyrus Hu, Edward Ramos, Fabio Tremblay, Jonathan Arora, Kanav Ellis, Kirsty Macesanu, Luca Villasevil, Marcel Torne Leonard, Matthew Cho, Meedeum Aslan, Ozgur Dass, Shivin Wang, Jie Reger, William Yuan, Xingfang Yang, Xuning Gupta, Abhishek Jayaraman, Dinesh Berseth, Glen Daniilidis, Kostas Martin-Martin, Roberto Lee, Youngwoon Liang, Percy Finn, Chelsea Levine, Sergey |
| contents | Comprehensive, unbiased, and comparable evaluation of modern generalist policies is uniquely challenging: existing approaches for robot benchmarking typically rely on heavy standardization, either by specifying fixed evaluation tasks and environments, or by hosting centralized ''robot challenges'', and do not readily scale to evaluating generalist policies across a broad range of tasks and environments. In this work, we propose RoboArena, a new approach for scalable evaluation of generalist robot policies in the real world. Instead of standardizing evaluations around fixed tasks, environments, or locations, we propose to crowd-source evaluations across a distributed network of evaluators. Importantly, evaluators can freely choose the tasks and environments they evaluate on, enabling easy scaling of diversity, but they are required to perform double-blind evaluations over pairs of policies. Then, by aggregating preference feedback from pairwise comparisons across diverse tasks and environments, we can derive a ranking of policies. We instantiate our approach across a network of evaluators at seven academic institutions using the DROID robot platform. Through more than 600 pairwise real-robot evaluation episodes across seven generalist policies, we demonstrate that our crowd-sourced approach can more accurately rank the performance of existing generalist policies than conventional, centralized evaluation approaches, while being more scalable, resilient, and trustworthy. We open our evaluation network to the community and hope that it can enable more accessible comparisons of generalist robot policies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_18123 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies Atreya, Pranav Pertsch, Karl Lee, Tony Kim, Moo Jin Jain, Arhan Kuramshin, Artur Eppner, Clemens Neary, Cyrus Hu, Edward Ramos, Fabio Tremblay, Jonathan Arora, Kanav Ellis, Kirsty Macesanu, Luca Villasevil, Marcel Torne Leonard, Matthew Cho, Meedeum Aslan, Ozgur Dass, Shivin Wang, Jie Reger, William Yuan, Xingfang Yang, Xuning Gupta, Abhishek Jayaraman, Dinesh Berseth, Glen Daniilidis, Kostas Martin-Martin, Roberto Lee, Youngwoon Liang, Percy Finn, Chelsea Levine, Sergey Robotics Machine Learning Comprehensive, unbiased, and comparable evaluation of modern generalist policies is uniquely challenging: existing approaches for robot benchmarking typically rely on heavy standardization, either by specifying fixed evaluation tasks and environments, or by hosting centralized ''robot challenges'', and do not readily scale to evaluating generalist policies across a broad range of tasks and environments. In this work, we propose RoboArena, a new approach for scalable evaluation of generalist robot policies in the real world. Instead of standardizing evaluations around fixed tasks, environments, or locations, we propose to crowd-source evaluations across a distributed network of evaluators. Importantly, evaluators can freely choose the tasks and environments they evaluate on, enabling easy scaling of diversity, but they are required to perform double-blind evaluations over pairs of policies. Then, by aggregating preference feedback from pairwise comparisons across diverse tasks and environments, we can derive a ranking of policies. We instantiate our approach across a network of evaluators at seven academic institutions using the DROID robot platform. Through more than 600 pairwise real-robot evaluation episodes across seven generalist policies, we demonstrate that our crowd-sourced approach can more accurately rank the performance of existing generalist policies than conventional, centralized evaluation approaches, while being more scalable, resilient, and trustworthy. We open our evaluation network to the community and hope that it can enable more accessible comparisons of generalist robot policies. |
| title | RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies |
| topic | Robotics Machine Learning |
| url | https://arxiv.org/abs/2506.18123 |