Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.13072 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917409329250304 |
|---|---|
| author | Long, Xiang Du, Li Xu, Yilong Liu, Fangcheng Wang, Haoqing Ding, Ning Li, Ziheng Guo, Jianyuan Tang, Yehui |
| author_facet | Long, Xiang Du, Li Xu, Yilong Liu, Fangcheng Wang, Haoqing Ding, Ning Li, Ziheng Guo, Jianyuan Tang, Yehui |
| contents | LLM-based agents are increasingly expected to handle real-world assistant tasks, yet existing benchmarks typically evaluate them under isolated sources of difficulty, such as a single environment or fully specified instructions. This leaves a substantial gap between current evaluation settings and the compositional challenges that arise in practical deployment. To address this gap, we introduce LiveClawBench, a benchmark to evaluate LLM agents on real-world assistant tasks. Based on an analysis of various real OpenClaw usage cases, we derive a Triple-Axis Complexity Framework that characterizes task difficulty along three dimensions: Environment Complexity, Cognitive Demand, and Runtime Adaptability. Guided by this framework, we construct a pilot benchmark with explicit complexity-factor annotations, covering real-world assistant tasks with compositional difficulty. Together, the framework and benchmark provide a principled foundation for evaluating LLM agents in realistic assistant settings, and establish a basis for future expansion across task domains and complexity axes. We are continuing to enrich our case collections to achieve more comprehensive domain and complexity coverage. The project page is at https://github.com/Mosi-AI/LiveClawBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_13072 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks Long, Xiang Du, Li Xu, Yilong Liu, Fangcheng Wang, Haoqing Ding, Ning Li, Ziheng Guo, Jianyuan Tang, Yehui Computation and Language Artificial Intelligence Machine Learning LLM-based agents are increasingly expected to handle real-world assistant tasks, yet existing benchmarks typically evaluate them under isolated sources of difficulty, such as a single environment or fully specified instructions. This leaves a substantial gap between current evaluation settings and the compositional challenges that arise in practical deployment. To address this gap, we introduce LiveClawBench, a benchmark to evaluate LLM agents on real-world assistant tasks. Based on an analysis of various real OpenClaw usage cases, we derive a Triple-Axis Complexity Framework that characterizes task difficulty along three dimensions: Environment Complexity, Cognitive Demand, and Runtime Adaptability. Guided by this framework, we construct a pilot benchmark with explicit complexity-factor annotations, covering real-world assistant tasks with compositional difficulty. Together, the framework and benchmark provide a principled foundation for evaluating LLM agents in realistic assistant settings, and establish a basis for future expansion across task domains and complexity axes. We are continuing to enrich our case collections to achieve more comprehensive domain and complexity coverage. The project page is at https://github.com/Mosi-AI/LiveClawBench. |
| title | LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2604.13072 |