RoboEval: Where Robotic Manipulation Meets Structured and Scalable Evaluation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909012864270336 |
|---|---|
| author | Wang, Yi Ru Ung, Carter Tan, Christopher Tannert, Grant Duan, Jiafei Li, Josephine Le, Anh Oswal, Rishabh Grotz, Markus Pumacay, Wilbert Deng, Yuquan Krishna, Ranjay Fox, Dieter Srinivasa, Siddhartha |
| author_facet | Wang, Yi Ru Ung, Carter Tan, Christopher Tannert, Grant Duan, Jiafei Li, Josephine Le, Anh Oswal, Rishabh Grotz, Markus Pumacay, Wilbert Deng, Yuquan Krishna, Ranjay Fox, Dieter Srinivasa, Siddhartha |
| contents | We introduce RoboEval, a structured evaluation framework and benchmark for robotic manipulation that augments binary success with principled behavioral and outcome metrics. Existing evaluations often collapse performance into outcome counts, masking differences in execution quality and obscuring failure structure. RoboEval provides eight bimanual tasks with systematically controlled variations, more than three thousand expert demonstrations, and a modular simulation platform for reproducible experimentation. All tasks are instrumented with standardized metrics that quantify efficiency, coordination, and safety/stability, as well as outcome measures that trace stagewise progress and localize failure modes. Through extensive experiments with state-of-the-art visuomotor policies, we validate these metrics by analyzing their stability under variation, discriminative power across policies with similar success rates, and correlation with task success. Project Page: https://robo-eval.github.io |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_00435 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RoboEval: Where Robotic Manipulation Meets Structured and Scalable Evaluation Wang, Yi Ru Ung, Carter Tan, Christopher Tannert, Grant Duan, Jiafei Li, Josephine Le, Anh Oswal, Rishabh Grotz, Markus Pumacay, Wilbert Deng, Yuquan Krishna, Ranjay Fox, Dieter Srinivasa, Siddhartha Robotics Artificial Intelligence Computer Vision and Pattern Recognition We introduce RoboEval, a structured evaluation framework and benchmark for robotic manipulation that augments binary success with principled behavioral and outcome metrics. Existing evaluations often collapse performance into outcome counts, masking differences in execution quality and obscuring failure structure. RoboEval provides eight bimanual tasks with systematically controlled variations, more than three thousand expert demonstrations, and a modular simulation platform for reproducible experimentation. All tasks are instrumented with standardized metrics that quantify efficiency, coordination, and safety/stability, as well as outcome measures that trace stagewise progress and localize failure modes. Through extensive experiments with state-of-the-art visuomotor policies, we validate these metrics by analyzing their stability under variation, discriminative power across policies with similar success rates, and correlation with task success. Project Page: https://robo-eval.github.io |
| title | RoboEval: Where Robotic Manipulation Meets Structured and Scalable Evaluation |
| topic | Robotics Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2507.00435 |