AnyTask: an Automated Task and Data Generation Framework for Advancing Sim-to-Real Policy Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914266990247936 |
|---|---|
| author | Gong, Ran Zhang, Xiaohan Shang, Jinghuan Minniti, Maria Vittoria Patel, Jigarkumar Pepe, Valerio Yan, Riedana Gundogdu, Ahmet Kapelyukh, Ivan Abbas, Ali Yan, Xiaoqiang Patel, Harsh Herlant, Laura Schmeckpeper, Karl |
| author_facet | Gong, Ran Zhang, Xiaohan Shang, Jinghuan Minniti, Maria Vittoria Patel, Jigarkumar Pepe, Valerio Yan, Riedana Gundogdu, Ahmet Kapelyukh, Ivan Abbas, Ali Yan, Xiaoqiang Patel, Harsh Herlant, Laura Schmeckpeper, Karl |
| contents | Generalist robot learning remains constrained by data: large-scale, diverse, and high-quality interaction data are expensive to collect in the real world. While simulation has become a promising way for scaling up data collection, the related tasks, including simulation task design, task-aware scene generation, expert demonstration synthesis, and sim-to-real transfer, still demand substantial human effort. We present AnyTask, an automated framework that pairs massively parallel GPU simulation with foundation models to design diverse manipulation tasks and synthesize robot data. We introduce three AnyTask agents for generating expert demonstrations aiming to solve as many tasks as possible: 1) ViPR, a novel task and motion planning agent with VLM-in-the-loop Parallel Refinement; 2) ViPR-Eureka, a reinforcement learning agent with generated dense rewards and LLM-guided contact sampling; 3) ViPR-RL, a hybrid planning and learning approach that jointly produces high-quality demonstrations with only sparse rewards. We train behavior cloning policies on generated data, validate them in simulation, and deploy them directly on real robot hardware. The policies generalize to novel object poses, achieving 44% average success across a suite of real-world pick-and-place, drawer opening, contact-rich pushing, and long-horizon manipulation tasks. Our project website is at https://anytask.rai-inst.com . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_17853 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AnyTask: an Automated Task and Data Generation Framework for Advancing Sim-to-Real Policy Learning Gong, Ran Zhang, Xiaohan Shang, Jinghuan Minniti, Maria Vittoria Patel, Jigarkumar Pepe, Valerio Yan, Riedana Gundogdu, Ahmet Kapelyukh, Ivan Abbas, Ali Yan, Xiaoqiang Patel, Harsh Herlant, Laura Schmeckpeper, Karl Robotics Artificial Intelligence Generalist robot learning remains constrained by data: large-scale, diverse, and high-quality interaction data are expensive to collect in the real world. While simulation has become a promising way for scaling up data collection, the related tasks, including simulation task design, task-aware scene generation, expert demonstration synthesis, and sim-to-real transfer, still demand substantial human effort. We present AnyTask, an automated framework that pairs massively parallel GPU simulation with foundation models to design diverse manipulation tasks and synthesize robot data. We introduce three AnyTask agents for generating expert demonstrations aiming to solve as many tasks as possible: 1) ViPR, a novel task and motion planning agent with VLM-in-the-loop Parallel Refinement; 2) ViPR-Eureka, a reinforcement learning agent with generated dense rewards and LLM-guided contact sampling; 3) ViPR-RL, a hybrid planning and learning approach that jointly produces high-quality demonstrations with only sparse rewards. We train behavior cloning policies on generated data, validate them in simulation, and deploy them directly on real robot hardware. The policies generalize to novel object poses, achieving 44% average success across a suite of real-world pick-and-place, drawer opening, contact-rich pushing, and long-horizon manipulation tasks. Our project website is at https://anytask.rai-inst.com . |
| title | AnyTask: an Automated Task and Data Generation Framework for Advancing Sim-to-Real Policy Learning |
| topic | Robotics Artificial Intelligence |
| url | https://arxiv.org/abs/2512.17853 |