LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912789725970432 |
|---|---|
| author | Fei, Senyu Wang, Siyin Shi, Junhao Dai, Zihao Cai, Jikun Qian, Pengfang Ji, Li He, Xinzhe Zhang, Shiduo Fei, Zhaoye Fu, Jinlan Gong, Jingjing Qiu, Xipeng |
| author_facet | Fei, Senyu Wang, Siyin Shi, Junhao Dai, Zihao Cai, Jikun Qian, Pengfang Ji, Li He, Xinzhe Zhang, Shiduo Fei, Zhaoye Fu, Jinlan Gong, Jingjing Qiu, Xipeng |
| contents | Visual-Language-Action (VLA) models report impressive success rates on robotic manipulation benchmarks, yet these results may mask fundamental weaknesses in robustness. We perform a systematic vulnerability analysis by introducing controlled perturbations across seven dimensions: objects layout, camera viewpoints, robot initial states, language instructions, light conditions, background textures and sensor noise. We comprehensively analyzed multiple state-of-the-art models and revealed consistent brittleness beneath apparent competence. Our analysis exposes critical weaknesses: models exhibit extreme sensitivity to perturbation factors, including camera viewpoints and robot initial states, with performance dropping from 95% to below 30% under modest perturbations. Surprisingly, models are largely insensitive to language variations, with further experiments revealing that models tend to ignore language instructions completely. Our findings challenge the assumption that high benchmark scores equate to true competency and highlight the need for evaluation practices that assess reliability under realistic variation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_13626 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models Fei, Senyu Wang, Siyin Shi, Junhao Dai, Zihao Cai, Jikun Qian, Pengfang Ji, Li He, Xinzhe Zhang, Shiduo Fei, Zhaoye Fu, Jinlan Gong, Jingjing Qiu, Xipeng Robotics Computation and Language Computer Vision and Pattern Recognition Visual-Language-Action (VLA) models report impressive success rates on robotic manipulation benchmarks, yet these results may mask fundamental weaknesses in robustness. We perform a systematic vulnerability analysis by introducing controlled perturbations across seven dimensions: objects layout, camera viewpoints, robot initial states, language instructions, light conditions, background textures and sensor noise. We comprehensively analyzed multiple state-of-the-art models and revealed consistent brittleness beneath apparent competence. Our analysis exposes critical weaknesses: models exhibit extreme sensitivity to perturbation factors, including camera viewpoints and robot initial states, with performance dropping from 95% to below 30% under modest perturbations. Surprisingly, models are largely insensitive to language variations, with further experiments revealing that models tend to ignore language instructions completely. Our findings challenge the assumption that high benchmark scores equate to true competency and highlight the need for evaluation practices that assess reliability under realistic variation. |
| title | LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models |
| topic | Robotics Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.13626 |