Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.04924 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917608213708800 |
|---|---|
| author | Li, Xiang Qiu, Kai Wang, Jinglu Xu, Xiaohao Singh, Rita Yamazak, Kashu Chen, Hao Huang, Xiaonan Raj, Bhiksha |
| author_facet | Li, Xiang Qiu, Kai Wang, Jinglu Xu, Xiaohao Singh, Rita Yamazak, Kashu Chen, Hao Huang, Xiaonan Raj, Bhiksha |
| contents | Referring perception, which aims at grounding visual objects with multimodal referring guidance, is essential for bridging the gap between humans, who provide instructions, and the environment where intelligent systems perceive. Despite progress in this field, the robustness of referring perception models (RPMs) against disruptive perturbations is not well explored. This work thoroughly assesses the resilience of RPMs against various perturbations in both general and specific contexts. Recognizing the complex nature of referring perception tasks, we present a comprehensive taxonomy of perturbations, and then develop a versatile toolbox for synthesizing and evaluating the effects of composite disturbances. Employing this toolbox, we construct $\text{R}^2$-Bench, a benchmark for assessing the Robustness of Referring perception models under noisy conditions across five key tasks. Moreover, we propose the $\text{R}^2$-Agent, an LLM-based agent that simplifies and automates model evaluation via natural language instructions. Our investigation uncovers the vulnerabilities of current RPMs to various perturbations and provides tools for assessing model robustness, potentially promoting the safe and resilient integration of intelligent systems into complex real-world scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_04924 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | $\text{R}^2$-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbations Li, Xiang Qiu, Kai Wang, Jinglu Xu, Xiaohao Singh, Rita Yamazak, Kashu Chen, Hao Huang, Xiaonan Raj, Bhiksha Computer Vision and Pattern Recognition Referring perception, which aims at grounding visual objects with multimodal referring guidance, is essential for bridging the gap between humans, who provide instructions, and the environment where intelligent systems perceive. Despite progress in this field, the robustness of referring perception models (RPMs) against disruptive perturbations is not well explored. This work thoroughly assesses the resilience of RPMs against various perturbations in both general and specific contexts. Recognizing the complex nature of referring perception tasks, we present a comprehensive taxonomy of perturbations, and then develop a versatile toolbox for synthesizing and evaluating the effects of composite disturbances. Employing this toolbox, we construct $\text{R}^2$-Bench, a benchmark for assessing the Robustness of Referring perception models under noisy conditions across five key tasks. Moreover, we propose the $\text{R}^2$-Agent, an LLM-based agent that simplifies and automates model evaluation via natural language instructions. Our investigation uncovers the vulnerabilities of current RPMs to various perturbations and provides tools for assessing model robustness, potentially promoting the safe and resilient integration of intelligent systems into complex real-world scenarios. |
| title | $\text{R}^2$-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbations |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.04924 |