Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.08119 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908847976742912 |
|---|---|
| author | Nandi, Subhrangshu Datta, Arghya Nama, Rohith Patel, Udita Vichare, Nikhil Bhattacharya, Indranil Grover, Prince Asija, Shivam Carenini, Giuseppe Zhang, Wei Gupta, Arushi Bhaduri, Sreyoshi Xu, Jing Raja, Huzefa Ray, Shayan Chan, Aaron Fei, Esther Xu Du, Gaoyuan Akhtar, Zuhaib Asnani, Harshita Chan, Weian Xiong, Ming Carbone, Francesco Mirchandani, Jeetu |
| author_facet | Nandi, Subhrangshu Datta, Arghya Nama, Rohith Patel, Udita Vichare, Nikhil Bhattacharya, Indranil Grover, Prince Asija, Shivam Carenini, Giuseppe Zhang, Wei Gupta, Arushi Bhaduri, Sreyoshi Xu, Jing Raja, Huzefa Ray, Shayan Chan, Aaron Fei, Esther Xu Du, Gaoyuan Akhtar, Zuhaib Asnani, Harshita Chan, Weian Xiong, Ming Carbone, Francesco Mirchandani, Jeetu |
| contents | LLM-based agents struggle to execute complex, multi-step Standard Operating Procedures (SOPs) that are fundamental to industrial automation. Existing benchmarks fail to capture the procedural complexity and tool orchestration demands of real-world workflows. We introduce SOP-Bench, a benchmark of 2,000+ tasks from human expert-authored SOPs across 12 business domains (healthcare, logistics, finance, content moderation, etc.). Using a human-AI collaborative framework, experts crafted authentic SOPs while AI generated artifacts (tools, APIs, datasets), all human-validated, yielding realistic tasks with executable interfaces and ground-truth outputs.
SOP-Bench serves as a research enabler for systematically investigating agent architectures, model capabilities, and deployment considerations across diverse procedural tasks. We demonstrate its utility through illustrative experiments with a subset of frontier models across Function-Calling (FC) and ReAct agents, revealing critical insights. For example, (1) newer models do not guarantee better performance - Claude 4 family outperforms Claude 4.5 family on ReAct tasks (Claude 4 Opus: 72.4% vs. Claude 4.5 Sonnet: 63.3% task success rate), demonstrating that production upgrades require validation; (2) no single model-agent combination dominates: best performances range from 57% to 100% depending on domain. These examples illustrate how SOP-Bench enables isolating and studying specific dimensions of agent performance without costly production experiments. Our goal is not to rank model capabilities or build optimal agents, but to provide a rigorous evaluation framework that enables the researchers and practitioners to systematically investigate agent design choices, model selection, and deployment strategies. We release the benchmark at https://github.com/amazon-science/sop-bench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08119 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agents Nandi, Subhrangshu Datta, Arghya Nama, Rohith Patel, Udita Vichare, Nikhil Bhattacharya, Indranil Grover, Prince Asija, Shivam Carenini, Giuseppe Zhang, Wei Gupta, Arushi Bhaduri, Sreyoshi Xu, Jing Raja, Huzefa Ray, Shayan Chan, Aaron Fei, Esther Xu Du, Gaoyuan Akhtar, Zuhaib Asnani, Harshita Chan, Weian Xiong, Ming Carbone, Francesco Mirchandani, Jeetu Artificial Intelligence LLM-based agents struggle to execute complex, multi-step Standard Operating Procedures (SOPs) that are fundamental to industrial automation. Existing benchmarks fail to capture the procedural complexity and tool orchestration demands of real-world workflows. We introduce SOP-Bench, a benchmark of 2,000+ tasks from human expert-authored SOPs across 12 business domains (healthcare, logistics, finance, content moderation, etc.). Using a human-AI collaborative framework, experts crafted authentic SOPs while AI generated artifacts (tools, APIs, datasets), all human-validated, yielding realistic tasks with executable interfaces and ground-truth outputs. SOP-Bench serves as a research enabler for systematically investigating agent architectures, model capabilities, and deployment considerations across diverse procedural tasks. We demonstrate its utility through illustrative experiments with a subset of frontier models across Function-Calling (FC) and ReAct agents, revealing critical insights. For example, (1) newer models do not guarantee better performance - Claude 4 family outperforms Claude 4.5 family on ReAct tasks (Claude 4 Opus: 72.4% vs. Claude 4.5 Sonnet: 63.3% task success rate), demonstrating that production upgrades require validation; (2) no single model-agent combination dominates: best performances range from 57% to 100% depending on domain. These examples illustrate how SOP-Bench enables isolating and studying specific dimensions of agent performance without costly production experiments. Our goal is not to rank model capabilities or build optimal agents, but to provide a rigorous evaluation framework that enables the researchers and practitioners to systematically investigate agent design choices, model selection, and deployment strategies. We release the benchmark at https://github.com/amazon-science/sop-bench. |
| title | SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agents |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2506.08119 |