On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915391584862208 |
|---|---|
| author | Zhao, Xingyu Aghazadeh-Chakherlou, Robab Cheng, Chih-Hong Popov, Peter Strigini, Lorenzo |
| author_facet | Zhao, Xingyu Aghazadeh-Chakherlou, Robab Cheng, Chih-Hong Popov, Peter Strigini, Lorenzo |
| contents | Scenario-based testing has emerged as a common method for autonomous vehicles (AVs) safety assessment, offering a more efficient alternative to mile-based testing by focusing on high-risk scenarios. However, fundamental questions persist regarding its stopping rules, residual risk estimation, debug effectiveness, and the impact of simulation fidelity on safety claims. This paper argues that a rigorous statistical foundation is essential to address these challenges and enable rigorous safety assurance. By drawing parallels between AV testing and established software testing methods, we identify shared research gaps and reusable solutions. We propose proof-of-concept models to quantify the probability of failure per scenario (\textit{pfs}) and evaluate testing effectiveness under varying conditions. Our analysis reveals that neither scenario-based nor mile-based testing universally outperforms the other. Furthermore, we give an example of formal reasoning about alignment of synthetic and real-world testing outcomes, a first step towards supporting statistically defensible simulation-based safety claims. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_02274 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles Zhao, Xingyu Aghazadeh-Chakherlou, Robab Cheng, Chih-Hong Popov, Peter Strigini, Lorenzo Software Engineering Artificial Intelligence Robotics Scenario-based testing has emerged as a common method for autonomous vehicles (AVs) safety assessment, offering a more efficient alternative to mile-based testing by focusing on high-risk scenarios. However, fundamental questions persist regarding its stopping rules, residual risk estimation, debug effectiveness, and the impact of simulation fidelity on safety claims. This paper argues that a rigorous statistical foundation is essential to address these challenges and enable rigorous safety assurance. By drawing parallels between AV testing and established software testing methods, we identify shared research gaps and reusable solutions. We propose proof-of-concept models to quantify the probability of failure per scenario (\textit{pfs}) and evaluate testing effectiveness under varying conditions. Our analysis reveals that neither scenario-based nor mile-based testing universally outperforms the other. Furthermore, we give an example of formal reasoning about alignment of synthetic and real-world testing outcomes, a first step towards supporting statistically defensible simulation-based safety claims. |
| title | On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles |
| topic | Software Engineering Artificial Intelligence Robotics |
| url | https://arxiv.org/abs/2505.02274 |