From Benchmarks to Business Impact: Deploying IBM Generalist Agent in Enterprise Production
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912755347357696 |
|---|---|
| author | Shlomov, Segev Oved, Alon Marreed, Sami Levy, Ido Akrabi, Offer Yaeli, Avi Strąk, Łukasz Koumpan, Elizabeth Goldshtein, Yinon Shapira, Eilam Mashkif, Nir Adi, Asaf |
| author_facet | Shlomov, Segev Oved, Alon Marreed, Sami Levy, Ido Akrabi, Offer Yaeli, Avi Strąk, Łukasz Koumpan, Elizabeth Goldshtein, Yinon Shapira, Eilam Mashkif, Nir Adi, Asaf |
| contents | Agents are rapidly advancing in automating digital work, but enterprises face a harder challenge: moving beyond prototypes to deployed systems that deliver measurable business value. This path is complicated by fragmented frameworks, slow development, and the absence of standardized evaluation practices. Generalist agents have emerged as a promising direction, excelling on academic benchmarks and offering flexibility across task types, applications, and modalities. Yet, evidence of their use in production enterprise settings remains limited. This paper reports IBM's experience developing and piloting the Computer Using Generalist Agent (CUGA), which has been open-sourced for the community (https://github.com/cuga-project/cuga-agent). CUGA adopts a hierarchical planner--executor architecture with strong analytical foundations, achieving state-of-the-art performance on AppWorld and WebArena. Beyond benchmarks, it was evaluated in a pilot within the Business-Process-Outsourcing talent acquisition domain, addressing enterprise requirements for scalability, auditability, safety, and governance. To support assessment, we introduce BPO-TA, a 26-task benchmark spanning 13 analytics endpoints. In preliminary evaluations, CUGA approached the accuracy of specialized agents while indicating potential for reducing development time and cost. Our contribution is twofold: presenting early evidence of generalist agents operating at enterprise scale, and distilling technical and organizational lessons from this initial pilot. We outline requirements and next steps for advancing research-grade architectures like CUGA into robust, enterprise-ready systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_23856 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Benchmarks to Business Impact: Deploying IBM Generalist Agent in Enterprise Production Shlomov, Segev Oved, Alon Marreed, Sami Levy, Ido Akrabi, Offer Yaeli, Avi Strąk, Łukasz Koumpan, Elizabeth Goldshtein, Yinon Shapira, Eilam Mashkif, Nir Adi, Asaf Artificial Intelligence 68Txx Agents are rapidly advancing in automating digital work, but enterprises face a harder challenge: moving beyond prototypes to deployed systems that deliver measurable business value. This path is complicated by fragmented frameworks, slow development, and the absence of standardized evaluation practices. Generalist agents have emerged as a promising direction, excelling on academic benchmarks and offering flexibility across task types, applications, and modalities. Yet, evidence of their use in production enterprise settings remains limited. This paper reports IBM's experience developing and piloting the Computer Using Generalist Agent (CUGA), which has been open-sourced for the community (https://github.com/cuga-project/cuga-agent). CUGA adopts a hierarchical planner--executor architecture with strong analytical foundations, achieving state-of-the-art performance on AppWorld and WebArena. Beyond benchmarks, it was evaluated in a pilot within the Business-Process-Outsourcing talent acquisition domain, addressing enterprise requirements for scalability, auditability, safety, and governance. To support assessment, we introduce BPO-TA, a 26-task benchmark spanning 13 analytics endpoints. In preliminary evaluations, CUGA approached the accuracy of specialized agents while indicating potential for reducing development time and cost. Our contribution is twofold: presenting early evidence of generalist agents operating at enterprise scale, and distilling technical and organizational lessons from this initial pilot. We outline requirements and next steps for advancing research-grade architectures like CUGA into robust, enterprise-ready systems. |
| title | From Benchmarks to Business Impact: Deploying IBM Generalist Agent in Enterprise Production |
| topic | Artificial Intelligence 68Txx |
| url | https://arxiv.org/abs/2510.23856 |