Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.11315 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918245803491328 |
|---|---|
| author | Guruprasad, Pranav Chowdhury, Sudipta Sikka, Harsh Sharma, Mridul Lu, Helen Rivera, Sean Khurana, Aryan Ren, Hangliang Wang, Yangyue |
| author_facet | Guruprasad, Pranav Chowdhury, Sudipta Sikka, Harsh Sharma, Mridul Lu, Helen Rivera, Sean Khurana, Aryan Ren, Hangliang Wang, Yangyue |
| contents | Generalist multimodal agents are expected to unify perception, language, and control - operating robustly across diverse real world domains. However, current evaluation practices remain fragmented across isolated benchmarks, making it difficult to assess whether today's foundation models truly generalize beyond their training distributions. We introduce MultiNet v1.0, a unified benchmark for measuring the cross domain generality of vision language models (VLMs) and vision language action models (VLAs) across six foundational capability regimes. Visual grounding, spatial reasoning, tool use, physical commonsense, multi agent coordination, and continuous robot control. Evaluating GPT 5, Pi0, and Magma, we find that no model demonstrates consistent generality. All exhibit substantial degradation on unseen domains, unfamiliar modalities, or cross domain task shifts despite strong performance within their training distributions.These failures manifest as modality misalignment, output format instability, and catastrophic knowledge degradation under domain transfer.Our findings reveal a persistent gap between the aspiration of generalist intelligence and the actual capabilities of current foundation models.MultiNet v1.0 provides a standardized evaluation substrate for diagnosing these gaps and guiding the development of future generalist agents.Code, data, and leaderboards are publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_11315 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Benchmarking the Generality of Vision-Language-Action Models Guruprasad, Pranav Chowdhury, Sudipta Sikka, Harsh Sharma, Mridul Lu, Helen Rivera, Sean Khurana, Aryan Ren, Hangliang Wang, Yangyue Machine Learning Generalist multimodal agents are expected to unify perception, language, and control - operating robustly across diverse real world domains. However, current evaluation practices remain fragmented across isolated benchmarks, making it difficult to assess whether today's foundation models truly generalize beyond their training distributions. We introduce MultiNet v1.0, a unified benchmark for measuring the cross domain generality of vision language models (VLMs) and vision language action models (VLAs) across six foundational capability regimes. Visual grounding, spatial reasoning, tool use, physical commonsense, multi agent coordination, and continuous robot control. Evaluating GPT 5, Pi0, and Magma, we find that no model demonstrates consistent generality. All exhibit substantial degradation on unseen domains, unfamiliar modalities, or cross domain task shifts despite strong performance within their training distributions.These failures manifest as modality misalignment, output format instability, and catastrophic knowledge degradation under domain transfer.Our findings reveal a persistent gap between the aspiration of generalist intelligence and the actual capabilities of current foundation models.MultiNet v1.0 provides a standardized evaluation substrate for diagnosing these gaps and guiding the development of future generalist agents.Code, data, and leaderboards are publicly available. |
| title | Benchmarking the Generality of Vision-Language-Action Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2512.11315 |