Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2604.23781 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913090293989376 |
|---|---|
| author | Meng, Fanqing Du, Lingxiao Wu, Zijian Chen, Guanzheng Liu, Xiangyan Liao, Jiaqi Jiang, Chonghe Wan, Zhenglin Gu, Jiawei Zhou, Pengfei Huang, Rui Zhao, Ziqi Ding, Shengyuan Yu, Ailing Peng, Bo Xia, Bowei Sun, Hao Liang, Haotian Xie, Ji Chen, Jiajun Song, Jiajun Yang, Liu Xu, Ming Qiu, Qionglin Fu, Runhao Zhai, Shengfang Wang, Shijian Ma, Tengfei Wu, Tianyi Jin, Weiyang Wang, Yan Dai, Yang Lai, Yao Shu, Youwei Liu, Yue Hao, Yunzhuo Niu, Yuwei Huang, Jinkai Zhuo, Jiayuan Shen, Zhennan Wu, Linyu Yao, Hannah Chen, Charles Xie, Cihang Zhou, Yuyin Zhang, Jiaheng Zheng, Zeyu Hu, Mengkang Shieh, Michael Qizhe |
| author_facet | Meng, Fanqing Du, Lingxiao Wu, Zijian Chen, Guanzheng Liu, Xiangyan Liao, Jiaqi Jiang, Chonghe Wan, Zhenglin Gu, Jiawei Zhou, Pengfei Huang, Rui Zhao, Ziqi Ding, Shengyuan Yu, Ailing Peng, Bo Xia, Bowei Sun, Hao Liang, Haotian Xie, Ji Chen, Jiajun Song, Jiajun Yang, Liu Xu, Ming Qiu, Qionglin Fu, Runhao Zhai, Shengfang Wang, Shijian Ma, Tengfei Wu, Tianyi Jin, Weiyang Wang, Yan Dai, Yang Lai, Yao Shu, Youwei Liu, Yue Hao, Yunzhuo Niu, Yuwei Huang, Jinkai Zhuo, Jiayuan Shen, Zhennan Wu, Linyu Yao, Hannah Chen, Charles Xie, Cihang Zhou, Yuyin Zhang, Jiaheng Zheng, Zeyu Hu, Mengkang Shieh, Michael Qizhe |
| contents | Language-model agents are increasingly used as persistent coworkers that assist users across multiple working days. During such workflows, the surrounding environment may change independently of the agent: new emails arrive, calendar entries shift, knowledge-base records are updated, and evidence appears across images, scanned PDFs, audio, video, and spreadsheets. Existing benchmarks do not adequately evaluate this setting because they typically run within a single static episode and remain largely text-centric. We introduce \bench{}, a benchmark for coworker agents built around multi-turn multi-day tasks, a stateful sandboxed service environment whose state evolves between turns, and rule-based verification. The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services (filesystem, email, calendar, knowledge base, spreadsheet) and scored by 1537 deterministic Python checkers over post-execution service state; no LLM-as-judge is invoked during scoring. We benchmark seven frontier agent systems. The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0\%, indicating that partial progress is common while complete end-to-end workflow completion remains rare. Turn-level analysis shows that performance drops after the first exogenous environment update, highlighting adaptation to changing state as a key open challenge. We release the benchmark, evaluation harness, and construction pipeline to support reproducible coworker-agent evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_23781 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents Meng, Fanqing Du, Lingxiao Wu, Zijian Chen, Guanzheng Liu, Xiangyan Liao, Jiaqi Jiang, Chonghe Wan, Zhenglin Gu, Jiawei Zhou, Pengfei Huang, Rui Zhao, Ziqi Ding, Shengyuan Yu, Ailing Peng, Bo Xia, Bowei Sun, Hao Liang, Haotian Xie, Ji Chen, Jiajun Song, Jiajun Yang, Liu Xu, Ming Qiu, Qionglin Fu, Runhao Zhai, Shengfang Wang, Shijian Ma, Tengfei Wu, Tianyi Jin, Weiyang Wang, Yan Dai, Yang Lai, Yao Shu, Youwei Liu, Yue Hao, Yunzhuo Niu, Yuwei Huang, Jinkai Zhuo, Jiayuan Shen, Zhennan Wu, Linyu Yao, Hannah Chen, Charles Xie, Cihang Zhou, Yuyin Zhang, Jiaheng Zheng, Zeyu Hu, Mengkang Shieh, Michael Qizhe Computer Vision and Pattern Recognition Software Engineering Language-model agents are increasingly used as persistent coworkers that assist users across multiple working days. During such workflows, the surrounding environment may change independently of the agent: new emails arrive, calendar entries shift, knowledge-base records are updated, and evidence appears across images, scanned PDFs, audio, video, and spreadsheets. Existing benchmarks do not adequately evaluate this setting because they typically run within a single static episode and remain largely text-centric. We introduce \bench{}, a benchmark for coworker agents built around multi-turn multi-day tasks, a stateful sandboxed service environment whose state evolves between turns, and rule-based verification. The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services (filesystem, email, calendar, knowledge base, spreadsheet) and scored by 1537 deterministic Python checkers over post-execution service state; no LLM-as-judge is invoked during scoring. We benchmark seven frontier agent systems. The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0\%, indicating that partial progress is common while complete end-to-end workflow completion remains rare. Turn-level analysis shows that performance drops after the first exogenous environment update, highlighting adaptation to changing state as a key open challenge. We release the benchmark, evaluation harness, and construction pipeline to support reproducible coworker-agent evaluation. |
| title | ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents |
| topic | Computer Vision and Pattern Recognition Software Engineering |
| url | https://arxiv.org/abs/2604.23781 |