A2Eval: Agentic and Automated Evaluation for Embodied Brain
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914300729229312 |
|---|---|
| author | Zhang, Shuai Hu, Jiayu Chen, Zijie Ding, Zeyuan Zhang, Yi Zhang, Yingji Zhou, Ziyi Liao, Junwei Zhou, Shengjie Dai, Yong Lan, Zhenzhong Ju, Xiaozhu |
| author_facet | Zhang, Shuai Hu, Jiayu Chen, Zijie Ding, Zeyuan Zhang, Yi Zhang, Yingji Zhou, Ziyi Liao, Junwei Zhou, Shengjie Dai, Yong Lan, Zhenzhong Ju, Xiaozhu |
| contents | Current embodied VLM evaluation relies on static, expert-defined, manually annotated benchmarks that exhibit severe redundancy and coverage imbalance. This labor intensive paradigm drains computational and annotation resources, inflates costs, and distorts model rankings, ultimately stifling iterative development. To address this, we propose Agentic Automatic Evaluation (A2Eval), the first agentic framework that automates benchmark curation and evaluation through two collaborative agents. The Data Agent autonomously induces capability dimensions and assembles a balanced, compact evaluation suite, while the Eval Agent synthesizes and validates executable evaluation pipelines, enabling fully autonomous, high-fidelity assessment. Evaluated across 10 benchmarks and 13 models, A2Eval compresses evaluation suites by 85%, reduces overall computational costs by 77%, and delivers a 4.6x speedup while preserving evaluation quality. Crucially, A2Eval corrects systematic ranking biases, improves human alignment to Spearman's rho=0.85, and maintains high ranking fidelity (Kendall's tau=0.81), establishing a new standard for high-fidelity, low-cost embodied assessment. Our code and data will be public soon. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_01640 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A2Eval: Agentic and Automated Evaluation for Embodied Brain Zhang, Shuai Hu, Jiayu Chen, Zijie Ding, Zeyuan Zhang, Yi Zhang, Yingji Zhou, Ziyi Liao, Junwei Zhou, Shengjie Dai, Yong Lan, Zhenzhong Ju, Xiaozhu Computation and Language Current embodied VLM evaluation relies on static, expert-defined, manually annotated benchmarks that exhibit severe redundancy and coverage imbalance. This labor intensive paradigm drains computational and annotation resources, inflates costs, and distorts model rankings, ultimately stifling iterative development. To address this, we propose Agentic Automatic Evaluation (A2Eval), the first agentic framework that automates benchmark curation and evaluation through two collaborative agents. The Data Agent autonomously induces capability dimensions and assembles a balanced, compact evaluation suite, while the Eval Agent synthesizes and validates executable evaluation pipelines, enabling fully autonomous, high-fidelity assessment. Evaluated across 10 benchmarks and 13 models, A2Eval compresses evaluation suites by 85%, reduces overall computational costs by 77%, and delivers a 4.6x speedup while preserving evaluation quality. Crucially, A2Eval corrects systematic ranking biases, improves human alignment to Spearman's rho=0.85, and maintains high ranking fidelity (Kendall's tau=0.81), establishing a new standard for high-fidelity, low-cost embodied assessment. Our code and data will be public soon. |
| title | A2Eval: Agentic and Automated Evaluation for Embodied Brain |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.01640 |