A Tool for Benchmarking Large Language Models' Robustness in Assessing the Realism of Driving Scenarios
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917064802828288 |
|---|---|
| author | Wu, Jiahui Lu, Chengjie Arrieta, Aitor Ali, Shaukat |
| author_facet | Wu, Jiahui Lu, Chengjie Arrieta, Aitor Ali, Shaukat |
| contents | In recent years, autonomous driving systems have made significant progress, yet ensuring their safety remains a key challenge. To this end, scenario-based testing offers a practical solution, and simulation-based methods have gained traction due to the high cost and risk of real-world testing. However, evaluating the realism of simulated scenarios remains difficult, creating demand for effective assessment methods. Recent advances show that Large Language Models (LLMs) possess strong reasoning and generalization capabilities, suggesting their potential in assessing scenario realism through scenario-related textual prompts. Motivated by this, we propose DriveRLR, a benchmark tool to assess the robustness of LLMs in evaluating the realism of driving scenarios. DriveRLR generates mutated scenario variants, constructs prompts, which are then used to assess a given LLM's ability and robustness in determining the realism of driving scenarios. We validate DriveRLR on the DeepScenario dataset using three state-of-the-art LLMs: GPT-5, Llama 4 Maverick, and Mistral Small 3.2. Results show that DriveRLR effectively reveals differences in the robustness of various LLMs, demonstrating its effectiveness and practical value in scenario realism assessment. Beyond LLM robustness evaluation, DriveRLR can serve as a practical component in applications such as an objective function to guide scenario generation, supporting simulation-based ADS testing workflows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_04267 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Tool for Benchmarking Large Language Models' Robustness in Assessing the Realism of Driving Scenarios Wu, Jiahui Lu, Chengjie Arrieta, Aitor Ali, Shaukat Software Engineering In recent years, autonomous driving systems have made significant progress, yet ensuring their safety remains a key challenge. To this end, scenario-based testing offers a practical solution, and simulation-based methods have gained traction due to the high cost and risk of real-world testing. However, evaluating the realism of simulated scenarios remains difficult, creating demand for effective assessment methods. Recent advances show that Large Language Models (LLMs) possess strong reasoning and generalization capabilities, suggesting their potential in assessing scenario realism through scenario-related textual prompts. Motivated by this, we propose DriveRLR, a benchmark tool to assess the robustness of LLMs in evaluating the realism of driving scenarios. DriveRLR generates mutated scenario variants, constructs prompts, which are then used to assess a given LLM's ability and robustness in determining the realism of driving scenarios. We validate DriveRLR on the DeepScenario dataset using three state-of-the-art LLMs: GPT-5, Llama 4 Maverick, and Mistral Small 3.2. Results show that DriveRLR effectively reveals differences in the robustness of various LLMs, demonstrating its effectiveness and practical value in scenario realism assessment. Beyond LLM robustness evaluation, DriveRLR can serve as a practical component in applications such as an objective function to guide scenario generation, supporting simulation-based ADS testing workflows. |
| title | A Tool for Benchmarking Large Language Models' Robustness in Assessing the Realism of Driving Scenarios |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2511.04267 |