AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914430930911232 |
|---|---|
| author | Huang, Tiancheng Cao, Ruisheng Zhang, Yuxin Kang, Zhangyi Wang, Zijian Wang, Chenrun Luo, Yijie Zheng, Hang Qian, Lirong Chen, Lu Yu, Kai |
| author_facet | Huang, Tiancheng Cao, Ruisheng Zhang, Yuxin Kang, Zhangyi Wang, Zijian Wang, Chenrun Luo, Yijie Zheng, Hang Qian, Lirong Chen, Lu Yu, Kai |
| contents | The growing volume of academic papers has made it increasingly difficult for researchers to efficiently extract key information. While large language models (LLMs) based agents are capable of automating question answering (QA) workflows for scientific papers, there still lacks a comprehensive and realistic benchmark to evaluate their capabilities. Moreover, training an interactive agent for this specific task is hindered by the shortage of high-quality interaction trajectories. In this work, we propose AirQA, a human-annotated comprehensive paper QA dataset in the field of artificial intelligence (AI), with 13,956 papers and 1,246 questions, that encompasses multi-task, multi-modal and instance-level evaluation. Furthermore, we propose ExTrActor, an automated framework for instruction data synthesis. With three LLM-based agents, ExTrActor can perform example generation and trajectory collection without human intervention. Evaluations of multiple open-source and proprietary models show that most models underperform on AirQA, demonstrating the quality of our dataset. Extensive experiments confirm that ExTrActor consistently improves the multi-turn tool-use capability of small models, enabling them to achieve performance comparable to larger ones. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_16952 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation Huang, Tiancheng Cao, Ruisheng Zhang, Yuxin Kang, Zhangyi Wang, Zijian Wang, Chenrun Luo, Yijie Zheng, Hang Qian, Lirong Chen, Lu Yu, Kai Computation and Language Artificial Intelligence The growing volume of academic papers has made it increasingly difficult for researchers to efficiently extract key information. While large language models (LLMs) based agents are capable of automating question answering (QA) workflows for scientific papers, there still lacks a comprehensive and realistic benchmark to evaluate their capabilities. Moreover, training an interactive agent for this specific task is hindered by the shortage of high-quality interaction trajectories. In this work, we propose AirQA, a human-annotated comprehensive paper QA dataset in the field of artificial intelligence (AI), with 13,956 papers and 1,246 questions, that encompasses multi-task, multi-modal and instance-level evaluation. Furthermore, we propose ExTrActor, an automated framework for instruction data synthesis. With three LLM-based agents, ExTrActor can perform example generation and trajectory collection without human intervention. Evaluations of multiple open-source and proprietary models show that most models underperform on AirQA, demonstrating the quality of our dataset. Extensive experiments confirm that ExTrActor consistently improves the multi-turn tool-use capability of small models, enabling them to achieve performance comparable to larger ones. |
| title | AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.16952 |