CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914431775014912 |
|---|---|
| author | Yu, Yi Hu, Guangquan Shen, Chenghuang Liu, Xingyan Gu, Jing Sun, Hangyi Ma, Junzhuo Liu, Weiting Liu, Jianfeng Pu, Mingyue Wang, Yu Xiao, Zhengdong Xie, Rui Luo, Longjiu Wang, Qianrong Cui, Gurong Qiao, Honglin Lu, Wenlian |
| author_facet | Yu, Yi Hu, Guangquan Shen, Chenghuang Liu, Xingyan Gu, Jing Sun, Hangyi Ma, Junzhuo Liu, Weiting Liu, Jianfeng Pu, Mingyue Wang, Yu Xiao, Zhengdong Xie, Rui Luo, Longjiu Wang, Qianrong Cui, Gurong Qiao, Honglin Lu, Wenlian |
| contents | The increasing agentic capabilities of Large Language Models (LLMs) have enabled their deployment in real-world applications, such as cloud services, where customer-assistant interactions exhibit high technical complexity and long-horizon dependencies, making robustness and resolution efficiency critical for customer satisfaction. However, existing benchmarks for LLM-based agents largely rely on synthetic environments that fail to capture the diversity and unpredictability of authentic customer inputs, often ignoring the resolution efficiency essential for real-world deployment. To bridge this gap, we introduce CirrusBench, a novel evaluation framework distinguished by its foundation in real-world data from authentic cloud service tickets. CirrusBench preserves the intricate multi-turn logical chains and realistic tool dependencies inherent to technical service environments. Moving beyond execution correctness, we introduce novel Customer-Centric metrics to define agent success, quantifying service quality through metrics such as the Normalized Efficiency Index and Multi-Turn Latency to explicitly measure resolution efficiency. Experiments utilizing our framework reveal that while state-of-the-art models demonstrate strong reasoning capabilities, they frequently struggle in complex, realistic multi-turn tasks and fail to meet the high-efficiency standards required for customer service, highlighting critical directions for the future development of LLM-based agents in practical technical service applications. CirrusBench evaluation framework is released at: https://github.com/CirrusAI |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_28569 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments Yu, Yi Hu, Guangquan Shen, Chenghuang Liu, Xingyan Gu, Jing Sun, Hangyi Ma, Junzhuo Liu, Weiting Liu, Jianfeng Pu, Mingyue Wang, Yu Xiao, Zhengdong Xie, Rui Luo, Longjiu Wang, Qianrong Cui, Gurong Qiao, Honglin Lu, Wenlian Machine Learning Artificial Intelligence Information Retrieval Performance The increasing agentic capabilities of Large Language Models (LLMs) have enabled their deployment in real-world applications, such as cloud services, where customer-assistant interactions exhibit high technical complexity and long-horizon dependencies, making robustness and resolution efficiency critical for customer satisfaction. However, existing benchmarks for LLM-based agents largely rely on synthetic environments that fail to capture the diversity and unpredictability of authentic customer inputs, often ignoring the resolution efficiency essential for real-world deployment. To bridge this gap, we introduce CirrusBench, a novel evaluation framework distinguished by its foundation in real-world data from authentic cloud service tickets. CirrusBench preserves the intricate multi-turn logical chains and realistic tool dependencies inherent to technical service environments. Moving beyond execution correctness, we introduce novel Customer-Centric metrics to define agent success, quantifying service quality through metrics such as the Normalized Efficiency Index and Multi-Turn Latency to explicitly measure resolution efficiency. Experiments utilizing our framework reveal that while state-of-the-art models demonstrate strong reasoning capabilities, they frequently struggle in complex, realistic multi-turn tasks and fail to meet the high-efficiency standards required for customer service, highlighting critical directions for the future development of LLM-based agents in practical technical service applications. CirrusBench evaluation framework is released at: https://github.com/CirrusAI |
| title | CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments |
| topic | Machine Learning Artificial Intelligence Information Retrieval Performance |
| url | https://arxiv.org/abs/2603.28569 |