ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916623930097664 |
|---|---|
| author | Zhou, Zhongyi Zhu, Yichen Zhu, Minjie Wen, Junjie Liu, Ning Xu, Zhiyuan Meng, Weibin Cheng, Ran Peng, Yaxin Shen, Chaomin Feng, Feifei |
| author_facet | Zhou, Zhongyi Zhu, Yichen Zhu, Minjie Wen, Junjie Liu, Ning Xu, Zhiyuan Meng, Weibin Cheng, Ran Peng, Yaxin Shen, Chaomin Feng, Feifei |
| contents | Humans possess a unified cognitive ability to perceive, comprehend, and interact with the physical world. Why can't large language models replicate this holistic understanding? Through a systematic analysis of existing training paradigms in vision-language-action models (VLA), we identify two key challenges: spurious forgetting, where robot training overwrites crucial visual-text alignments, and task interference, where competing control and understanding tasks degrade performance when trained jointly. To overcome these limitations, we propose ChatVLA, a novel framework featuring Phased Alignment Training, which incrementally integrates multimodal data after initial control mastery, and a Mixture-of-Experts architecture to minimize task interference. ChatVLA demonstrates competitive performance on visual question-answering datasets and significantly surpasses state-of-the-art vision-language-action (VLA) methods on multimodal understanding benchmarks. Notably, it achieves a six times higher performance on MMMU and scores 47.2% on MMStar with a more parameter-efficient design than ECoT. Furthermore, ChatVLA demonstrates superior performance on 25 real-world robot manipulation tasks compared to existing VLA methods like OpenVLA. Our findings highlight the potential of our unified framework for achieving both robust multimodal understanding and effective robot control. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_14420 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model Zhou, Zhongyi Zhu, Yichen Zhu, Minjie Wen, Junjie Liu, Ning Xu, Zhiyuan Meng, Weibin Cheng, Ran Peng, Yaxin Shen, Chaomin Feng, Feifei Robotics Computer Vision and Pattern Recognition Machine Learning Humans possess a unified cognitive ability to perceive, comprehend, and interact with the physical world. Why can't large language models replicate this holistic understanding? Through a systematic analysis of existing training paradigms in vision-language-action models (VLA), we identify two key challenges: spurious forgetting, where robot training overwrites crucial visual-text alignments, and task interference, where competing control and understanding tasks degrade performance when trained jointly. To overcome these limitations, we propose ChatVLA, a novel framework featuring Phased Alignment Training, which incrementally integrates multimodal data after initial control mastery, and a Mixture-of-Experts architecture to minimize task interference. ChatVLA demonstrates competitive performance on visual question-answering datasets and significantly surpasses state-of-the-art vision-language-action (VLA) methods on multimodal understanding benchmarks. Notably, it achieves a six times higher performance on MMMU and scores 47.2% on MMStar with a more parameter-efficient design than ECoT. Furthermore, ChatVLA demonstrates superior performance on 25 real-world robot manipulation tasks compared to existing VLA methods like OpenVLA. Our findings highlight the potential of our unified framework for achieving both robust multimodal understanding and effective robot control. |
| title | ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model |
| topic | Robotics Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2502.14420 |