GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908406053339136 |
|---|---|
| author | Gao, Ning Chen, Yilun Yang, Shuai Chen, Xinyi Tian, Yang Li, Hao Huang, Haifeng Wang, Hanqing Wang, Tai Pang, Jiangmiao |
| author_facet | Gao, Ning Chen, Yilun Yang, Shuai Chen, Xinyi Tian, Yang Li, Hao Huang, Haifeng Wang, Hanqing Wang, Tai Pang, Jiangmiao |
| contents | Robotic manipulation in real-world settings remains challenging, especially regarding robust generalization. Existing simulation platforms lack sufficient support for exploring how policies adapt to varied instructions and scenarios. Thus, they lag behind the growing interest in instruction-following foundation models like LLMs, whose adaptability is crucial yet remains underexplored in fair comparisons. To bridge this gap, we introduce GenManip, a realistic tabletop simulation platform tailored for policy generalization studies. It features an automatic pipeline via LLM-driven task-oriented scene graph to synthesize large-scale, diverse tasks using 10K annotated 3D object assets. To systematically assess generalization, we present GenManip-Bench, a benchmark of 200 scenarios refined via human-in-the-loop corrections. We evaluate two policy types: (1) modular manipulation systems integrating foundation models for perception, reasoning, and planning, and (2) end-to-end policies trained through scalable data collection. Results show that while data scaling benefits end-to-end methods, modular systems enhanced with foundation models generalize more effectively across diverse scenarios. We anticipate this platform to facilitate critical insights for advancing policy generalization in realistic conditions. Project Page: https://genmanip.axi404.top/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_10966 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation Gao, Ning Chen, Yilun Yang, Shuai Chen, Xinyi Tian, Yang Li, Hao Huang, Haifeng Wang, Hanqing Wang, Tai Pang, Jiangmiao Robotics Robotic manipulation in real-world settings remains challenging, especially regarding robust generalization. Existing simulation platforms lack sufficient support for exploring how policies adapt to varied instructions and scenarios. Thus, they lag behind the growing interest in instruction-following foundation models like LLMs, whose adaptability is crucial yet remains underexplored in fair comparisons. To bridge this gap, we introduce GenManip, a realistic tabletop simulation platform tailored for policy generalization studies. It features an automatic pipeline via LLM-driven task-oriented scene graph to synthesize large-scale, diverse tasks using 10K annotated 3D object assets. To systematically assess generalization, we present GenManip-Bench, a benchmark of 200 scenarios refined via human-in-the-loop corrections. We evaluate two policy types: (1) modular manipulation systems integrating foundation models for perception, reasoning, and planning, and (2) end-to-end policies trained through scalable data collection. Results show that while data scaling benefits end-to-end methods, modular systems enhanced with foundation models generalize more effectively across diverse scenarios. We anticipate this platform to facilitate critical insights for advancing policy generalization in realistic conditions. Project Page: https://genmanip.axi404.top/. |
| title | GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation |
| topic | Robotics |
| url | https://arxiv.org/abs/2506.10966 |