Copertina

ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Jha, Saurabh, Arora, Rohan, Watanabe, Yuji, Yanagawa, Takumi, Chen, Yinfang, Clark, Jackson, Bhavya, Bhavya, Verma, Mudit, Kumar, Harshit, Kitahara, Hirokuni, Zheutlin, Noah, Takano, Saki, Pathak, Divya, George, Felix, Wu, Xinbo, Turkkan, Bekir O., Vanloo, Gerard, Nidd, Michael, Dai, Ting, Chatterjee, Oishik, Gupta, Pranjal, Samanta, Suranjana, Aggarwal, Pooja, Lee, Rong, Murali, Pavankumar, Ahn, Jae-wook, Kar, Debanjana, Rahane, Ameet, Fonseca, Carlos, Paradkar, Amit, Deng, Yu, Moogi, Pratibha, Mohapatra, Prateeti, Abe, Naoki, Narayanaswami, Chandrasekhar, Xu, Tianyin, Varshney, Lav R., Mahindru, Ruchi, Sailer, Anca, Shwartz, Laura, Sow, Daby, Fuller, Nicholas C. M., Puri, Ruchir
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Artificial Intelligence Distributed, Parallel, and Cluster Computing Multiagent Systems
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

_version_	1866910819055304704
author	Jha, Saurabh Arora, Rohan Watanabe, Yuji Yanagawa, Takumi Chen, Yinfang Clark, Jackson Bhavya, Bhavya Verma, Mudit Kumar, Harshit Kitahara, Hirokuni Zheutlin, Noah Takano, Saki Pathak, Divya George, Felix Wu, Xinbo Turkkan, Bekir O. Vanloo, Gerard Nidd, Michael Dai, Ting Chatterjee, Oishik Gupta, Pranjal Samanta, Suranjana Aggarwal, Pooja Lee, Rong Murali, Pavankumar Ahn, Jae-wook Kar, Debanjana Rahane, Ameet Fonseca, Carlos Paradkar, Amit Deng, Yu Moogi, Pratibha Mohapatra, Prateeti Abe, Naoki Narayanaswami, Chandrasekhar Xu, Tianyin Varshney, Lav R. Mahindru, Ruchi Sailer, Anca Shwartz, Laura Sow, Daby Fuller, Nicholas C. M. Puri, Ruchir
author_facet	Jha, Saurabh Arora, Rohan Watanabe, Yuji Yanagawa, Takumi Chen, Yinfang Clark, Jackson Bhavya, Bhavya Verma, Mudit Kumar, Harshit Kitahara, Hirokuni Zheutlin, Noah Takano, Saki Pathak, Divya George, Felix Wu, Xinbo Turkkan, Bekir O. Vanloo, Gerard Nidd, Michael Dai, Ting Chatterjee, Oishik Gupta, Pranjal Samanta, Suranjana Aggarwal, Pooja Lee, Rong Murali, Pavankumar Ahn, Jae-wook Kar, Debanjana Rahane, Ameet Fonseca, Carlos Paradkar, Amit Deng, Yu Moogi, Pratibha Mohapatra, Prateeti Abe, Naoki Narayanaswami, Chandrasekhar Xu, Tianyin Varshney, Lav R. Mahindru, Ruchi Sailer, Anca Shwartz, Laura Sow, Daby Fuller, Nicholas C. M. Puri, Ruchir
contents	Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. ITBench includes an initial set of 94 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps scenarios. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast.
format	Preprint
id	arxiv_https___arxiv_org_abs_2502_05352
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks Jha, Saurabh Arora, Rohan Watanabe, Yuji Yanagawa, Takumi Chen, Yinfang Clark, Jackson Bhavya, Bhavya Verma, Mudit Kumar, Harshit Kitahara, Hirokuni Zheutlin, Noah Takano, Saki Pathak, Divya George, Felix Wu, Xinbo Turkkan, Bekir O. Vanloo, Gerard Nidd, Michael Dai, Ting Chatterjee, Oishik Gupta, Pranjal Samanta, Suranjana Aggarwal, Pooja Lee, Rong Murali, Pavankumar Ahn, Jae-wook Kar, Debanjana Rahane, Ameet Fonseca, Carlos Paradkar, Amit Deng, Yu Moogi, Pratibha Mohapatra, Prateeti Abe, Naoki Narayanaswami, Chandrasekhar Xu, Tianyin Varshney, Lav R. Mahindru, Ruchi Sailer, Anca Shwartz, Laura Sow, Daby Fuller, Nicholas C. M. Puri, Ruchir Artificial Intelligence Distributed, Parallel, and Cluster Computing Multiagent Systems Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. ITBench includes an initial set of 94 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps scenarios. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast.
title	ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks
topic	Artificial Intelligence Distributed, Parallel, and Cluster Computing Multiagent Systems
url	https://arxiv.org/abs/2502.05352