Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.15636 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917364139819008 |
|---|---|
| author | Yang, Yibo Lei, Fei Sun, Yixuan Zeng, Yantao Lv, Chengguang Hong, Jiancao Tian, Jiaojiao Qiu, Tianyu Wang, Xin Chen, Yanbing Li, Yanjie Pan, Zheng Zhou, Xiaochen Chen, Guanzhou Lv, Haoran Xu, Yuning Ou, Yue Liu, Haodong He, Shiqi Jia, Anya Xin, Yulei Wu, Huan Liu, Liang Ge, Jiaye Dong, Jianxin Lin, Dahua Sun, Wenxiu |
| author_facet | Yang, Yibo Lei, Fei Sun, Yixuan Zeng, Yantao Lv, Chengguang Hong, Jiancao Tian, Jiaojiao Qiu, Tianyu Wang, Xin Chen, Yanbing Li, Yanjie Pan, Zheng Zhou, Xiaochen Chen, Guanzhou Lv, Haoran Xu, Yuning Ou, Yue Liu, Haodong He, Shiqi Jia, Anya Xin, Yulei Wu, Huan Liu, Liang Ge, Jiaye Dong, Jianxin Lin, Dahua Sun, Wenxiu |
| contents | As AI-driven document understanding and processing tools become increasingly prevalent in real-world applications, the need for rigorous evaluation standards has grown increasingly urgent. Existing benchmarks and evaluations often focus on isolated capabilities or simplified scenarios, failing to capture the end-to-end task effectiveness required in practical settings. To address this gap, we introduce AIDABench, a comprehensive benchmark for evaluating AI systems on complex data analytics tasks in an end-to-end manner. AIDABench encompasses 600+ diverse document analysis tasks across three core capability dimensions: question answering, data visualization, and file generation. These tasks are grounded in realistic scenarios involving heterogeneous data types, including spreadsheets, databases, financial reports, and operational records, and reflect analytical demands across diverse industries and job functions. Notably, the tasks in AIDABench are sufficiently challenging that even human experts require 1-2 hours per question when assisted by AI tools, underscoring the benchmark's difficulty and real-world complexity. We evaluate 11 state-of-the-art models on AIDABench, spanning both proprietary (e.g., Claude Sonnet 4.5, Gemini 3 Pro Preview) and open-source (e.g., Qwen3-Max-2026-01-23-Thinking) families. Our results reveal that complex, real-world data analytics tasks remain a significant challenge for current AI systems, with the best-performing model achieving only 59.43% pass-at-1. We provide a detailed analysis of failure modes across each capability dimension and identify key challenges for future research. AIDABench offers a principled reference for enterprise procurement, tool selection, and model optimization, and is publicly available at https://github.com/MichaelYang-lyx/AIDABench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_15636 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AIDABench: AI Data Analytics Benchmark Yang, Yibo Lei, Fei Sun, Yixuan Zeng, Yantao Lv, Chengguang Hong, Jiancao Tian, Jiaojiao Qiu, Tianyu Wang, Xin Chen, Yanbing Li, Yanjie Pan, Zheng Zhou, Xiaochen Chen, Guanzhou Lv, Haoran Xu, Yuning Ou, Yue Liu, Haodong He, Shiqi Jia, Anya Xin, Yulei Wu, Huan Liu, Liang Ge, Jiaye Dong, Jianxin Lin, Dahua Sun, Wenxiu Artificial Intelligence As AI-driven document understanding and processing tools become increasingly prevalent in real-world applications, the need for rigorous evaluation standards has grown increasingly urgent. Existing benchmarks and evaluations often focus on isolated capabilities or simplified scenarios, failing to capture the end-to-end task effectiveness required in practical settings. To address this gap, we introduce AIDABench, a comprehensive benchmark for evaluating AI systems on complex data analytics tasks in an end-to-end manner. AIDABench encompasses 600+ diverse document analysis tasks across three core capability dimensions: question answering, data visualization, and file generation. These tasks are grounded in realistic scenarios involving heterogeneous data types, including spreadsheets, databases, financial reports, and operational records, and reflect analytical demands across diverse industries and job functions. Notably, the tasks in AIDABench are sufficiently challenging that even human experts require 1-2 hours per question when assisted by AI tools, underscoring the benchmark's difficulty and real-world complexity. We evaluate 11 state-of-the-art models on AIDABench, spanning both proprietary (e.g., Claude Sonnet 4.5, Gemini 3 Pro Preview) and open-source (e.g., Qwen3-Max-2026-01-23-Thinking) families. Our results reveal that complex, real-world data analytics tasks remain a significant challenge for current AI systems, with the best-performing model achieving only 59.43% pass-at-1. We provide a detailed analysis of failure modes across each capability dimension and identify key challenges for future research. AIDABench offers a principled reference for enterprise procurement, tool selection, and model optimization, and is publicly available at https://github.com/MichaelYang-lyx/AIDABench. |
| title | AIDABench: AI Data Analytics Benchmark |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2603.15636 |