PRM-as-a-Judge: A Dense Evaluation Paradigm for Fine-Grained Robotic Auditing
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917357837877248 |
|---|---|
| author | Ji, Yuheng Liu, Yuyang Tan, Huajie Huang, Xuchuan Huang, Fanding Xu, Yijie Chi, Cheng Zhao, Yuting Lyu, Huaihai Co, Peterson Cao, Mingyu Zhang, Qiongyu Li, Zhe Zhou, Enshen Wang, Pengwei Wang, Zhongyuan Zhang, Shanghang Zheng, Xiaolong |
| author_facet | Ji, Yuheng Liu, Yuyang Tan, Huajie Huang, Xuchuan Huang, Fanding Xu, Yijie Chi, Cheng Zhao, Yuting Lyu, Huaihai Co, Peterson Cao, Mingyu Zhang, Qiongyu Li, Zhe Zhou, Enshen Wang, Pengwei Wang, Zhongyuan Zhang, Shanghang Zheng, Xiaolong |
| contents | Current robotic evaluation is still largely dominated by binary success rates, which collapse rich execution processes into a single outcome and obscure critical qualities such as progress, efficiency, and stability. To address this limitation, we propose PRM-as-a-Judge, a dense evaluation paradigm that leverages Process Reward Models (PRMs) to audit policy execution directly from trajectory videos by estimating task progress from observation sequences. Central to this paradigm is the OPD (Outcome-Process-Diagnosis) metric system, which explicitly formalizes execution quality via a task-aligned progress potential. We characterize dense robotic evaluation through two axiomatic properties: macro-consistency, which requires additive and path-consistent aggregation, and micro-resolution, which requires sensitivity to fine-grained physical evolution. Under this formulation, potential-based PRM judges provide a natural instantiation of dense evaluation, with macro-consistency following directly from the induced scalar potential. We empirically validate the micro-resolution property using RoboPulse, a diagnostic benchmark specifically designed for probing micro-scale progress discrimination, where several trajectory-trained PRM judges outperform discriminative similarity-based methods and general-purpose foundation-model judges. Finally, leveraging PRM-as-a-Judge and the OPD metric system, we conduct a structured audit of mainstream policy paradigms across long-horizon tasks, revealing behavioral signatures and failure modes that are invisible to outcome-only metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_21669 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PRM-as-a-Judge: A Dense Evaluation Paradigm for Fine-Grained Robotic Auditing Ji, Yuheng Liu, Yuyang Tan, Huajie Huang, Xuchuan Huang, Fanding Xu, Yijie Chi, Cheng Zhao, Yuting Lyu, Huaihai Co, Peterson Cao, Mingyu Zhang, Qiongyu Li, Zhe Zhou, Enshen Wang, Pengwei Wang, Zhongyuan Zhang, Shanghang Zheng, Xiaolong Robotics Computer Vision and Pattern Recognition Current robotic evaluation is still largely dominated by binary success rates, which collapse rich execution processes into a single outcome and obscure critical qualities such as progress, efficiency, and stability. To address this limitation, we propose PRM-as-a-Judge, a dense evaluation paradigm that leverages Process Reward Models (PRMs) to audit policy execution directly from trajectory videos by estimating task progress from observation sequences. Central to this paradigm is the OPD (Outcome-Process-Diagnosis) metric system, which explicitly formalizes execution quality via a task-aligned progress potential. We characterize dense robotic evaluation through two axiomatic properties: macro-consistency, which requires additive and path-consistent aggregation, and micro-resolution, which requires sensitivity to fine-grained physical evolution. Under this formulation, potential-based PRM judges provide a natural instantiation of dense evaluation, with macro-consistency following directly from the induced scalar potential. We empirically validate the micro-resolution property using RoboPulse, a diagnostic benchmark specifically designed for probing micro-scale progress discrimination, where several trajectory-trained PRM judges outperform discriminative similarity-based methods and general-purpose foundation-model judges. Finally, leveraging PRM-as-a-Judge and the OPD metric system, we conduct a structured audit of mainstream policy paradigms across long-horizon tasks, revealing behavioral signatures and failure modes that are invisible to outcome-only metrics. |
| title | PRM-as-a-Judge: A Dense Evaluation Paradigm for Fine-Grained Robotic Auditing |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.21669 |