CABENCH: Benchmarking Composable AI for Solving Complex Tasks through Composing Ready-to-Use Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913974675570688 |
|---|---|
| author | Pham, Tung-Thuy Luong, Duy-Quan Duong, Minh-Quan Nguyen, Trung-Hieu Nguyen, Thu-Trang Nguyen, Son Vo, Hieu Dinh |
| author_facet | Pham, Tung-Thuy Luong, Duy-Quan Duong, Minh-Quan Nguyen, Trung-Hieu Nguyen, Thu-Trang Nguyen, Son Vo, Hieu Dinh |
| contents | Composable AI offers a scalable and effective paradigm for tackling complex AI tasks by decomposing them into sub-tasks and solving each sub-task using ready-to-use well-trained models. However, systematically evaluating methods under this setting remains largely unexplored. In this paper, we introduce CABENCH, the first public benchmark comprising 70 realistic composable AI tasks, along with a curated pool of 700 models across multiple modalities and domains. We also propose an evaluation framework to enable end-to-end assessment of composable AI solutions. To establish initial baselines, we provide human-designed reference solutions and compare their performance with two LLM-based approaches. Our results illustrate the promise of composable AI in addressing complex real-world problems while highlighting the need for methods that can fully unlock its potential by automatically generating effective execution pipelines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_02427 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CABENCH: Benchmarking Composable AI for Solving Complex Tasks through Composing Ready-to-Use Models Pham, Tung-Thuy Luong, Duy-Quan Duong, Minh-Quan Nguyen, Trung-Hieu Nguyen, Thu-Trang Nguyen, Son Vo, Hieu Dinh Artificial Intelligence Software Engineering Composable AI offers a scalable and effective paradigm for tackling complex AI tasks by decomposing them into sub-tasks and solving each sub-task using ready-to-use well-trained models. However, systematically evaluating methods under this setting remains largely unexplored. In this paper, we introduce CABENCH, the first public benchmark comprising 70 realistic composable AI tasks, along with a curated pool of 700 models across multiple modalities and domains. We also propose an evaluation framework to enable end-to-end assessment of composable AI solutions. To establish initial baselines, we provide human-designed reference solutions and compare their performance with two LLM-based approaches. Our results illustrate the promise of composable AI in addressing complex real-world problems while highlighting the need for methods that can fully unlock its potential by automatically generating effective execution pipelines. |
| title | CABENCH: Benchmarking Composable AI for Solving Complex Tasks through Composing Ready-to-Use Models |
| topic | Artificial Intelligence Software Engineering |
| url | https://arxiv.org/abs/2508.02427 |