MM-ACT: Learn from Multimodal Parallel Generation to Act
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909948968960000 |
|---|---|
| author | Liang, Haotian Chen, Xinyi Wang, Bin Chen, Mingkang Liu, Yitian Zhang, Yuhao Chen, Zanxin Yang, Tianshuo Chen, Yilun Pang, Jiangmiao Liu, Dong Yang, Xiaokang Mu, Yao Shao, Wenqi Luo, Ping |
| author_facet | Liang, Haotian Chen, Xinyi Wang, Bin Chen, Mingkang Liu, Yitian Zhang, Yuhao Chen, Zanxin Yang, Tianshuo Chen, Yilun Pang, Jiangmiao Liu, Dong Yang, Xiaokang Mu, Yao Shao, Wenqi Luo, Ping |
| contents | A generalist robotic policy needs both semantic understanding for task planning and the ability to interact with the environment through predictive capabilities. To tackle this, we present MM-ACT, a unified Vision-Language-Action (VLA) model that integrates text, image, and action in shared token space and performs generation across all three modalities. MM-ACT adopts a re-mask parallel decoding strategy for text and image generation, and employs a one-step parallel decoding strategy for action generation to improve efficiency. We introduce Context-Shared Multimodal Learning, a unified training paradigm that supervises generation in all three modalities from a shared context, enhancing action generation through cross-modal learning. Experiments were conducted on the LIBERO simulation and Franka real-robot setups as well as RoboTwin2.0 to assess in-domain and out-of-domain performances respectively. Our approach achieves a success rate of 96.3% on LIBERO, 72.0% across three tasks of real Franka, and 52.38% across eight bimanual tasks of RoboTwin2.0 with an additional gain of 9.25% from cross-modal learning. We release our codes, models and data at https://github.com/HHYHRHY/MM-ACT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_00975 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MM-ACT: Learn from Multimodal Parallel Generation to Act Liang, Haotian Chen, Xinyi Wang, Bin Chen, Mingkang Liu, Yitian Zhang, Yuhao Chen, Zanxin Yang, Tianshuo Chen, Yilun Pang, Jiangmiao Liu, Dong Yang, Xiaokang Mu, Yao Shao, Wenqi Luo, Ping Computer Vision and Pattern Recognition Machine Learning Robotics A generalist robotic policy needs both semantic understanding for task planning and the ability to interact with the environment through predictive capabilities. To tackle this, we present MM-ACT, a unified Vision-Language-Action (VLA) model that integrates text, image, and action in shared token space and performs generation across all three modalities. MM-ACT adopts a re-mask parallel decoding strategy for text and image generation, and employs a one-step parallel decoding strategy for action generation to improve efficiency. We introduce Context-Shared Multimodal Learning, a unified training paradigm that supervises generation in all three modalities from a shared context, enhancing action generation through cross-modal learning. Experiments were conducted on the LIBERO simulation and Franka real-robot setups as well as RoboTwin2.0 to assess in-domain and out-of-domain performances respectively. Our approach achieves a success rate of 96.3% on LIBERO, 72.0% across three tasks of real Franka, and 52.38% across eight bimanual tasks of RoboTwin2.0 with an additional gain of 9.25% from cross-modal learning. We release our codes, models and data at https://github.com/HHYHRHY/MM-ACT. |
| title | MM-ACT: Learn from Multimodal Parallel Generation to Act |
| topic | Computer Vision and Pattern Recognition Machine Learning Robotics |
| url | https://arxiv.org/abs/2512.00975 |