Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.13285 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915917183582208 |
|---|---|
| author | Romanou, Angelika Ibrahim, Mark Ross, Candace Shaib, Chantal Oktar, Kerem Bell, Samuel J. Ovalle, Anaelia Dodge, Jesse Bosselut, Antoine Sinha, Koustuv Williams, Adina |
| author_facet | Romanou, Angelika Ibrahim, Mark Ross, Candace Shaib, Chantal Oktar, Kerem Bell, Samuel J. Ovalle, Anaelia Dodge, Jesse Bosselut, Antoine Sinha, Koustuv Williams, Adina |
| contents | Existing evaluation methods largely rely on clean, static benchmarks, which can overestimate true model performance by failing to capture the noise and variability inherent in real-world user inputs. This is especially true for language models, which can face human-generated text queries containing mistakes, typos, or alternative ways of phrasing the same question. In this work, we introduce a theoretical framework for quantifying model sensitivity to prompt variants, or brittleness, that can enable us to disentangle data-induced difficulty from prompt-related variability. Using this framework, we design a novel evaluation pipeline, Brittlebench, to holistically evaluate the sensitivity of frontier models. We apply semantics-preserving perturbations to a suite of popular benchmarks, and observe model performance to degrade as much as 12%. However, these perturbations do not affect all models equally: even a single perturbation alters the relative ranking of models in 63% of cases, impacting conclusions about comparative model performance. Decomposing the total variance of both state-of-the-art open-weight and commercial models, we find that semantics-preserving input perturbations can account for up to half of the performance variance for a given model. Brittlebench highlights the need for more robust evaluations and models, and allows us to systematically understand model brittleness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_13285 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Brittlebench: Quantifying LLM robustness via prompt sensitivity Romanou, Angelika Ibrahim, Mark Ross, Candace Shaib, Chantal Oktar, Kerem Bell, Samuel J. Ovalle, Anaelia Dodge, Jesse Bosselut, Antoine Sinha, Koustuv Williams, Adina Machine Learning Artificial Intelligence I.2 Existing evaluation methods largely rely on clean, static benchmarks, which can overestimate true model performance by failing to capture the noise and variability inherent in real-world user inputs. This is especially true for language models, which can face human-generated text queries containing mistakes, typos, or alternative ways of phrasing the same question. In this work, we introduce a theoretical framework for quantifying model sensitivity to prompt variants, or brittleness, that can enable us to disentangle data-induced difficulty from prompt-related variability. Using this framework, we design a novel evaluation pipeline, Brittlebench, to holistically evaluate the sensitivity of frontier models. We apply semantics-preserving perturbations to a suite of popular benchmarks, and observe model performance to degrade as much as 12%. However, these perturbations do not affect all models equally: even a single perturbation alters the relative ranking of models in 63% of cases, impacting conclusions about comparative model performance. Decomposing the total variance of both state-of-the-art open-weight and commercial models, we find that semantics-preserving input perturbations can account for up to half of the performance variance for a given model. Brittlebench highlights the need for more robust evaluations and models, and allows us to systematically understand model brittleness. |
| title | Brittlebench: Quantifying LLM robustness via prompt sensitivity |
| topic | Machine Learning Artificial Intelligence I.2 |
| url | https://arxiv.org/abs/2603.13285 |