Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.14262 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914476761022464 |
|---|---|
| author | Wang, Yangyue Sikka, Harshvardhan Mathur, Yash Zhou, Tony Nyachhyon, Jinu Guruprasad, Pranav |
| author_facet | Wang, Yangyue Sikka, Harshvardhan Mathur, Yash Zhou, Tony Nyachhyon, Jinu Guruprasad, Pranav |
| contents | GUI grounding models report over 85% accuracy on standard benchmarks, yet drop 27-56 percentage points when instructions require spatial reasoning rather than direct element naming. Current benchmarks miss this because they evaluate each screenshot once with a single fixed instruction. We introduce GUI-Perturbed, a controlled perturbation framework that independently varies visual scenes and instructions to measure grounding robustness. Evaluating three 7B models from the same architecture lineage, we find that relational instructions cause systematic accuracy collapse across all models, a 70% browser zoom produces statistically significant degradation, and rank-8 LoRA fine-tuning with augmented data degrades performance rather than improving it. By perturbing along independent axes, GUI-Perturbed isolates which specific capability axes are affected-spatial reasoning, visual robustness, reasoning calibration-providing diagnostic signal that aggregate benchmarks cannot. We release the dataset, augmentation pipeline, and a fine-tuned model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_14262 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models Wang, Yangyue Sikka, Harshvardhan Mathur, Yash Zhou, Tony Nyachhyon, Jinu Guruprasad, Pranav Machine Learning Artificial Intelligence GUI grounding models report over 85% accuracy on standard benchmarks, yet drop 27-56 percentage points when instructions require spatial reasoning rather than direct element naming. Current benchmarks miss this because they evaluate each screenshot once with a single fixed instruction. We introduce GUI-Perturbed, a controlled perturbation framework that independently varies visual scenes and instructions to measure grounding robustness. Evaluating three 7B models from the same architecture lineage, we find that relational instructions cause systematic accuracy collapse across all models, a 70% browser zoom produces statistically significant degradation, and rank-8 LoRA fine-tuning with augmented data degrades performance rather than improving it. By perturbing along independent axes, GUI-Perturbed isolates which specific capability axes are affected-spatial reasoning, visual robustness, reasoning calibration-providing diagnostic signal that aggregate benchmarks cannot. We release the dataset, augmentation pipeline, and a fine-tuned model. |
| title | GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2604.14262 |