Sim2Val: Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916931600121856 |
|---|---|
| author | Luo, Rachel Yang, Heng Watson, Michael Sharma, Apoorva Veer, Sushant Schmerling, Edward Pavone, Marco |
| author_facet | Luo, Rachel Yang, Heng Watson, Michael Sharma, Apoorva Veer, Sushant Schmerling, Edward Pavone, Marco |
| contents | Learning-based robotic systems demand rigorous validation to assure reliable performance, but extensive real-world testing is often prohibitively expensive, and if conducted may still yield insufficient data for high-confidence guarantees. In this work we introduce Sim2Val, a general estimation framework that leverages paired data across test platforms, e.g., paired simulation and real-world observations, to achieve better estimates of real-world metrics via the method of control variates. By incorporating cheap and abundant auxiliary measurements (for example, simulator outputs) as control variates for costly real-world samples, our method provably reduces the variance of Monte Carlo estimates and thus requires significantly fewer real-world samples to attain a specified confidence bound on the mean performance. We provide theoretical analysis characterizing the variance and sample-efficiency improvement, and demonstrate empirically in autonomous driving and quadruped robotics settings that our approach achieves high-probability bounds with markedly improved sample efficiency. Our technique can lower the real-world testing burden for validating the performance of the stack, thereby enabling more efficient and cost-effective experimental evaluation of robotic systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_20553 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Sim2Val: Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation Luo, Rachel Yang, Heng Watson, Michael Sharma, Apoorva Veer, Sushant Schmerling, Edward Pavone, Marco Robotics Learning-based robotic systems demand rigorous validation to assure reliable performance, but extensive real-world testing is often prohibitively expensive, and if conducted may still yield insufficient data for high-confidence guarantees. In this work we introduce Sim2Val, a general estimation framework that leverages paired data across test platforms, e.g., paired simulation and real-world observations, to achieve better estimates of real-world metrics via the method of control variates. By incorporating cheap and abundant auxiliary measurements (for example, simulator outputs) as control variates for costly real-world samples, our method provably reduces the variance of Monte Carlo estimates and thus requires significantly fewer real-world samples to attain a specified confidence bound on the mean performance. We provide theoretical analysis characterizing the variance and sample-efficiency improvement, and demonstrate empirically in autonomous driving and quadruped robotics settings that our approach achieves high-probability bounds with markedly improved sample efficiency. Our technique can lower the real-world testing burden for validating the performance of the stack, thereby enabling more efficient and cost-effective experimental evaluation of robotic systems. |
| title | Sim2Val: Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation |
| topic | Robotics |
| url | https://arxiv.org/abs/2506.20553 |