Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911656399863808 |
|---|---|
| author | Ye, Bowen Li, Rang Yang, Qibin Liu, Yuanxin Yao, Linli Lv, Hanglong Xie, Zhihui An, Chenxin Li, Lei Kong, Lingpeng Liu, Qi Sui, Zhifang Yang, Tong |
| author_facet | Ye, Bowen Li, Rang Yang, Qibin Liu, Yuanxin Yao, Linli Lv, Hanglong Xie, Zhihui An, Chenxin Li, Lei Kong, Lingpeng Liu, Qi Sui, Zhifang Yang, Tong |
| contents | Large language models are increasingly deployed as autonomous agents for multi-step workflows in real-world software environments. However, existing agent benchmarks are limited by trajectory-opaque grading, underspecified safety and robustness evaluation, and narrow coverage of modalities and interaction paradigms. We introduce Claw-Eval, an end-to-end evaluation suite addressing these gaps with 300 human-verified tasks spanning 9 categories across three groups: general service orchestration, multimodal perception and interaction, and multi-turn professional dialogue. To enable trajectory-aware grading, each run is recorded through three independent evidence channels: execution traces, audit logs, and environment snapshots, yielding 2,159 fine-grained rubric items. The scoring protocol evaluates Completion, Safety, and Robustness, with Average Score, Pass@k, and Pass^k across three trials to distinguish genuine capability from lucky outcomes. Experiments on 14 frontier models show that: (1) Trajectory-opaque evaluation is systematically unreliable, missing 44% of safety violations and 13% of robustness failures detected by our framework. (2) Capability does not imply consistency, with Pass@3 remaining stable under error injection while Pass^3 dropping by up to 24 percentage points. (3) Agent capability is strongly multi-dimensional, with model rankings varying across task groups and metrics, indicating that our heterogeneous evaluation coverage is essential. Claw-Eval highlights directions for developing agents that are not only capable but reliably deployable. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_06132 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents Ye, Bowen Li, Rang Yang, Qibin Liu, Yuanxin Yao, Linli Lv, Hanglong Xie, Zhihui An, Chenxin Li, Lei Kong, Lingpeng Liu, Qi Sui, Zhifang Yang, Tong Artificial Intelligence Large language models are increasingly deployed as autonomous agents for multi-step workflows in real-world software environments. However, existing agent benchmarks are limited by trajectory-opaque grading, underspecified safety and robustness evaluation, and narrow coverage of modalities and interaction paradigms. We introduce Claw-Eval, an end-to-end evaluation suite addressing these gaps with 300 human-verified tasks spanning 9 categories across three groups: general service orchestration, multimodal perception and interaction, and multi-turn professional dialogue. To enable trajectory-aware grading, each run is recorded through three independent evidence channels: execution traces, audit logs, and environment snapshots, yielding 2,159 fine-grained rubric items. The scoring protocol evaluates Completion, Safety, and Robustness, with Average Score, Pass@k, and Pass^k across three trials to distinguish genuine capability from lucky outcomes. Experiments on 14 frontier models show that: (1) Trajectory-opaque evaluation is systematically unreliable, missing 44% of safety violations and 13% of robustness failures detected by our framework. (2) Capability does not imply consistency, with Pass@3 remaining stable under error injection while Pass^3 dropping by up to 24 percentage points. (3) Agent capability is strongly multi-dimensional, with model rankings varying across task groups and metrics, indicating that our heterogeneous evaluation coverage is essential. Claw-Eval highlights directions for developing agents that are not only capable but reliably deployable. |
| title | Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2604.06132 |