Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.20613 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908799689818112 |
|---|---|
| author | Chen, Kaiyuan Wu, Qimin Hou, Taiyu Tang, Tianhao Hu, Xueyu Hou, Yuchen Li, Bikun Qian, Chengming Wang, Guoyin Chen, Haolin Tian, Haotong Zhang, Haoye Bian, Haoyu Pan, Hongbing Zhang, Hongkang Zhou, Hongyi Cai, Jiaqi Rao, Jiewu Ren, Jiyuan Huang, Keduan Huang, Lucia Zhu Yuan, Mingyu Guo, Naixu Tang, Qicheng Zhang, Qinyan Chen, Shuai Chen, Siheng Li, Ting Ting Guo, Xiaoxing Zuo, Yaocheng Guo, Yaoqi Wang, Yinan Yu, Yinzhou Wang, Yize Jiang, Yuan Tian, Yuan Zhang, Yuanshuo Liu, Yuxuan Zeng, Yvette Yan Shan, Zenyu Yin, Zihan Hu, Xiaobo Liu, Yang Ren, Yixin Gong, Yuan |
| author_facet | Chen, Kaiyuan Wu, Qimin Hou, Taiyu Tang, Tianhao Hu, Xueyu Hou, Yuchen Li, Bikun Qian, Chengming Wang, Guoyin Chen, Haolin Tian, Haotong Zhang, Haoye Bian, Haoyu Pan, Hongbing Zhang, Hongkang Zhou, Hongyi Cai, Jiaqi Rao, Jiewu Ren, Jiyuan Huang, Keduan Huang, Lucia Zhu Yuan, Mingyu Guo, Naixu Tang, Qicheng Zhang, Qinyan Chen, Shuai Chen, Siheng Li, Ting Ting Guo, Xiaoxing Zuo, Yaocheng Guo, Yaoqi Wang, Yinan Yu, Yinzhou Wang, Yize Jiang, Yuan Tian, Yuan Zhang, Yuanshuo Liu, Yuxuan Zeng, Yvette Yan Shan, Zenyu Yin, Zihan Hu, Xiaobo Liu, Yang Ren, Yixin Gong, Yuan |
| contents | The capacity of AI agents to effectively handle tasks of increasing duration and complexity continues to grow, demonstrating exceptional performance in coding, deep research, and complex problem-solving evaluations. However, in daily scenarios, the perception of these advanced AI capabilities among general users remains limited. We argue that current evaluations prioritize increasing task difficulty without sufficiently addressing the diversity of agentic tasks necessary to cover the daily work, life, and learning activities of a broad demographic. To address this, we propose AgentIF-OneDay, aimed at determining whether general users can utilize natural language instructions and AI agents to complete a diverse array of daily tasks. These tasks require not only solving problems through dialogue but also understanding various attachment types and delivering tangible file-based results. The benchmark is structured around three user-centric categories: Open Workflow Execution, which assesses adherence to explicit and complex workflows; Latent Instruction, which requires agents to infer implicit instructions from attachments; and Iterative Refinement, which involves modifying or expanding upon ongoing work. We employ instance-level rubrics and a refined evaluation pipeline that aligns LLM-based verification with human judgment, achieving an 80.1% agreement rate using Gemini-3-Pro. AgentIF-OneDay comprises 104 tasks covering 767 scoring points. We benchmarked four leading general AI agents and found that agent products built based on APIs and ChatGPT agents based on agent RL remain in the first tier simultaneously. Leading LLM APIs and open-source models have internalized agentic capabilities, enabling AI application teams to develop cutting-edge Agent products. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_20613 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios Chen, Kaiyuan Wu, Qimin Hou, Taiyu Tang, Tianhao Hu, Xueyu Hou, Yuchen Li, Bikun Qian, Chengming Wang, Guoyin Chen, Haolin Tian, Haotong Zhang, Haoye Bian, Haoyu Pan, Hongbing Zhang, Hongkang Zhou, Hongyi Cai, Jiaqi Rao, Jiewu Ren, Jiyuan Huang, Keduan Huang, Lucia Zhu Yuan, Mingyu Guo, Naixu Tang, Qicheng Zhang, Qinyan Chen, Shuai Chen, Siheng Li, Ting Ting Guo, Xiaoxing Zuo, Yaocheng Guo, Yaoqi Wang, Yinan Yu, Yinzhou Wang, Yize Jiang, Yuan Tian, Yuan Zhang, Yuanshuo Liu, Yuxuan Zeng, Yvette Yan Shan, Zenyu Yin, Zihan Hu, Xiaobo Liu, Yang Ren, Yixin Gong, Yuan Computation and Language The capacity of AI agents to effectively handle tasks of increasing duration and complexity continues to grow, demonstrating exceptional performance in coding, deep research, and complex problem-solving evaluations. However, in daily scenarios, the perception of these advanced AI capabilities among general users remains limited. We argue that current evaluations prioritize increasing task difficulty without sufficiently addressing the diversity of agentic tasks necessary to cover the daily work, life, and learning activities of a broad demographic. To address this, we propose AgentIF-OneDay, aimed at determining whether general users can utilize natural language instructions and AI agents to complete a diverse array of daily tasks. These tasks require not only solving problems through dialogue but also understanding various attachment types and delivering tangible file-based results. The benchmark is structured around three user-centric categories: Open Workflow Execution, which assesses adherence to explicit and complex workflows; Latent Instruction, which requires agents to infer implicit instructions from attachments; and Iterative Refinement, which involves modifying or expanding upon ongoing work. We employ instance-level rubrics and a refined evaluation pipeline that aligns LLM-based verification with human judgment, achieving an 80.1% agreement rate using Gemini-3-Pro. AgentIF-OneDay comprises 104 tasks covering 767 scoring points. We benchmarked four leading general AI agents and found that agent products built based on APIs and ChatGPT agents based on agent RL remain in the first tier simultaneously. Leading LLM APIs and open-source models have internalized agentic capabilities, enabling AI application teams to develop cutting-edge Agent products. |
| title | AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.20613 |