LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instruction
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908557706788864 |
|---|---|
| author | Xue, Haoru Huang, Xiaoyu Niu, Dantong Liao, Qiayuan Kragerud, Thomas Gravdahl, Jan Tommy Peng, Xue Bin Shi, Guanya Darrell, Trevor Sreenath, Koushil Sastry, Shankar |
| author_facet | Xue, Haoru Huang, Xiaoyu Niu, Dantong Liao, Qiayuan Kragerud, Thomas Gravdahl, Jan Tommy Peng, Xue Bin Shi, Guanya Darrell, Trevor Sreenath, Koushil Sastry, Shankar |
| contents | Vision-language-action (VLA) models have demonstrated strong semantic understanding and zero-shot generalization, yet most existing systems assume an accurate low-level controller with hand-crafted action "vocabulary" such as end-effector pose or root velocity. This assumption confines prior work to quasi-static tasks and precludes the agile, whole-body behaviors required by humanoid whole-body control (WBC) tasks. To capture this gap in the literature, we start by introducing the first sim-to-real-ready, vision-language, closed-loop benchmark for humanoid WBC, comprising over 150 tasks from 10 categories. We then propose LeVERB: Latent Vision-Language-Encoded Robot Behavior, a hierarchical latent instruction-following framework for humanoid vision-language WBC, the first of its kind. At the top level, a vision-language policy learns a latent action vocabulary from synthetically rendered kinematic demonstrations; at the low level, a reinforcement-learned WBC policy consumes these latent verbs to generate dynamics-level commands. In our benchmark, LeVERB can zero-shot attain a 80% success rate on simple visual navigation tasks, and 58.5% success rate overall, outperforming naive hierarchical whole-body VLA implementation by 7.8 times. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_13751 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instruction Xue, Haoru Huang, Xiaoyu Niu, Dantong Liao, Qiayuan Kragerud, Thomas Gravdahl, Jan Tommy Peng, Xue Bin Shi, Guanya Darrell, Trevor Sreenath, Koushil Sastry, Shankar Robotics Artificial Intelligence Vision-language-action (VLA) models have demonstrated strong semantic understanding and zero-shot generalization, yet most existing systems assume an accurate low-level controller with hand-crafted action "vocabulary" such as end-effector pose or root velocity. This assumption confines prior work to quasi-static tasks and precludes the agile, whole-body behaviors required by humanoid whole-body control (WBC) tasks. To capture this gap in the literature, we start by introducing the first sim-to-real-ready, vision-language, closed-loop benchmark for humanoid WBC, comprising over 150 tasks from 10 categories. We then propose LeVERB: Latent Vision-Language-Encoded Robot Behavior, a hierarchical latent instruction-following framework for humanoid vision-language WBC, the first of its kind. At the top level, a vision-language policy learns a latent action vocabulary from synthetically rendered kinematic demonstrations; at the low level, a reinforcement-learned WBC policy consumes these latent verbs to generate dynamics-level commands. In our benchmark, LeVERB can zero-shot attain a 80% success rate on simple visual navigation tasks, and 58.5% success rate overall, outperforming naive hierarchical whole-body VLA implementation by 7.8 times. |
| title | LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instruction |
| topic | Robotics Artificial Intelligence |
| url | https://arxiv.org/abs/2506.13751 |