Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2605.02661 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917458730811392 |
|---|---|
| author | Yu, Junjie Lu, Pengrui Si, Weiye Lu, Hongliang Wu, Jiabao Tao, Kaiwen Wang, Kun Yang, Lingyu Zhang, Qiran Guo, Xiuting Wang, Xuanyu Wang, Yang Wang, Yanjie Yang, Yi Hu, Zijian Yang, Ziyi Zhou, Zonghan Qiang, Binghao Zhang, Borui Li, Chenning Zhang, Enchang Chen, Feifan Jian, Feng Sun, Fengyin Qiu, Hao Zheng, Hao Zhu, Haoran Liu, Hongyu Deng, Jianbin Song, Jiaxin Chi, Jiaying Shi, Jiayou Fang, Jie Zhong, Jinghui Zhou, Jingyu Li, Jinze Yi, Junfeng Yu, Junyan Xue, Junzhi Song, Ni Chen, Pengyi Chen, Qi Li, Quansheng Tao, Rui Gong, Shenghai Lu, Shenhang Shen, Tianqi Zhu, Tianxiang Kang, Tiehan Li, Tingyu Wu, Wendi Shen, Xiao Zhou, Xiao Zhang, Xiaotao Li, Xinrong Yang, Xuankun Zhang, Xun Li, Yan Lu, Ye Wang, Yi Zhou, Yibo Zhang, Yichi Sun, Yihao Huang, Yijun Zhu, Yixin Wu, Yixuan Sun, Yuchen Wu, Yue Sun, Yuheng Li, Yukun Tu, Yutian Qin, Yuxuan Wu, Yuzhuo Li, Zeyu Lou, Zhengyu Ran, Zhenning He, Zizhu Liu, Pengfei |
| author_facet | Yu, Junjie Lu, Pengrui Si, Weiye Lu, Hongliang Wu, Jiabao Tao, Kaiwen Wang, Kun Yang, Lingyu Zhang, Qiran Guo, Xiuting Wang, Xuanyu Wang, Yang Wang, Yanjie Yang, Yi Hu, Zijian Yang, Ziyi Zhou, Zonghan Qiang, Binghao Zhang, Borui Li, Chenning Zhang, Enchang Chen, Feifan Jian, Feng Sun, Fengyin Qiu, Hao Zheng, Hao Zhu, Haoran Liu, Hongyu Deng, Jianbin Song, Jiaxin Chi, Jiaying Shi, Jiayou Fang, Jie Zhong, Jinghui Zhou, Jingyu Li, Jinze Yi, Junfeng Yu, Junyan Xue, Junzhi Song, Ni Chen, Pengyi Chen, Qi Li, Quansheng Tao, Rui Gong, Shenghai Lu, Shenhang Shen, Tianqi Zhu, Tianxiang Kang, Tiehan Li, Tingyu Wu, Wendi Shen, Xiao Zhou, Xiao Zhang, Xiaotao Li, Xinrong Yang, Xuankun Zhang, Xun Li, Yan Lu, Ye Wang, Yi Zhou, Yibo Zhang, Yichi Sun, Yihao Huang, Yijun Zhu, Yixin Wu, Yixuan Sun, Yuchen Wu, Yue Sun, Yuheng Li, Yukun Tu, Yutian Qin, Yuxuan Wu, Yuzhuo Li, Zeyu Lou, Zhengyu Ran, Zhenning He, Zizhu Liu, Pengfei |
| contents | Benchmarks within the OpenClaw ecosystem have thus far evaluated exclusively assistant-level tasks, leaving the academic-level capabilities of OpenClaw largely unexamined. We introduce AcademiClaw, a bilingual benchmark of 80 complex, long-horizon tasks sourced directly from university students' real academic workflows -- homework, research projects, competitions, and personal projects -- that they found current AI agents unable to solve effectively. Curated from 230 student-submitted candidates through rigorous expert review, the final task set spans 25+ professional domains, ranging from olympiad-level mathematics and linguistics problems to GPU-intensive reinforcement learning and full-stack system debugging, with 16 tasks requiring CUDA GPU execution. Each task executes in an isolated Docker sandbox and is scored on task completion by multi-dimensional rubrics combining six complementary techniques, with an independent five-category safety audit providing additional behavioral analysis. Experiments on six frontier models show that even the best achieves only a 55\% pass rate. Further analysis uncovers sharp capability boundaries across task domains, divergent behavioral strategies among models, and a disconnect between token consumption and output quality, providing fine-grained diagnostic signals beyond what aggregate metrics reveal. We hope that AcademiClaw and its open-sourced data and code can serve as a useful resource for the OpenClaw community, driving progress toward agents that are more capable and versatile across the full breadth of real-world academic demands. All data and code are available at https://github.com/GAIR-NLP/AcademiClaw. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_02661 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AcademiClaw: When Students Set Challenges for AI Agents Yu, Junjie Lu, Pengrui Si, Weiye Lu, Hongliang Wu, Jiabao Tao, Kaiwen Wang, Kun Yang, Lingyu Zhang, Qiran Guo, Xiuting Wang, Xuanyu Wang, Yang Wang, Yanjie Yang, Yi Hu, Zijian Yang, Ziyi Zhou, Zonghan Qiang, Binghao Zhang, Borui Li, Chenning Zhang, Enchang Chen, Feifan Jian, Feng Sun, Fengyin Qiu, Hao Zheng, Hao Zhu, Haoran Liu, Hongyu Deng, Jianbin Song, Jiaxin Chi, Jiaying Shi, Jiayou Fang, Jie Zhong, Jinghui Zhou, Jingyu Li, Jinze Yi, Junfeng Yu, Junyan Xue, Junzhi Song, Ni Chen, Pengyi Chen, Qi Li, Quansheng Tao, Rui Gong, Shenghai Lu, Shenhang Shen, Tianqi Zhu, Tianxiang Kang, Tiehan Li, Tingyu Wu, Wendi Shen, Xiao Zhou, Xiao Zhang, Xiaotao Li, Xinrong Yang, Xuankun Zhang, Xun Li, Yan Lu, Ye Wang, Yi Zhou, Yibo Zhang, Yichi Sun, Yihao Huang, Yijun Zhu, Yixin Wu, Yixuan Sun, Yuchen Wu, Yue Sun, Yuheng Li, Yukun Tu, Yutian Qin, Yuxuan Wu, Yuzhuo Li, Zeyu Lou, Zhengyu Ran, Zhenning He, Zizhu Liu, Pengfei Artificial Intelligence Computers and Society Benchmarks within the OpenClaw ecosystem have thus far evaluated exclusively assistant-level tasks, leaving the academic-level capabilities of OpenClaw largely unexamined. We introduce AcademiClaw, a bilingual benchmark of 80 complex, long-horizon tasks sourced directly from university students' real academic workflows -- homework, research projects, competitions, and personal projects -- that they found current AI agents unable to solve effectively. Curated from 230 student-submitted candidates through rigorous expert review, the final task set spans 25+ professional domains, ranging from olympiad-level mathematics and linguistics problems to GPU-intensive reinforcement learning and full-stack system debugging, with 16 tasks requiring CUDA GPU execution. Each task executes in an isolated Docker sandbox and is scored on task completion by multi-dimensional rubrics combining six complementary techniques, with an independent five-category safety audit providing additional behavioral analysis. Experiments on six frontier models show that even the best achieves only a 55\% pass rate. Further analysis uncovers sharp capability boundaries across task domains, divergent behavioral strategies among models, and a disconnect between token consumption and output quality, providing fine-grained diagnostic signals beyond what aggregate metrics reveal. We hope that AcademiClaw and its open-sourced data and code can serve as a useful resource for the OpenClaw community, driving progress toward agents that are more capable and versatile across the full breadth of real-world academic demands. All data and code are available at https://github.com/GAIR-NLP/AcademiClaw. |
| title | AcademiClaw: When Students Set Challenges for AI Agents |
| topic | Artificial Intelligence Computers and Society |
| url | https://arxiv.org/abs/2605.02661 |