LangProBe: a Language Programs Benchmark
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915175802601472 |
|---|---|
| author | Tan, Shangyin Agrawal, Lakshya A Singhvi, Arnav Lai, Liheng Ryan, Michael J Klein, Dan Khattab, Omar Sen, Koushik Zaharia, Matei |
| author_facet | Tan, Shangyin Agrawal, Lakshya A Singhvi, Arnav Lai, Liheng Ryan, Michael J Klein, Dan Khattab, Omar Sen, Koushik Zaharia, Matei |
| contents | Composing language models (LMs) into multi-step language programs and automatically optimizing their modular prompts is now a mainstream paradigm for building AI systems, but the tradeoffs in this space have only scarcely been studied before. We introduce LangProBe, the first large-scale benchmark for evaluating the architectures and optimization strategies for language programs, with over 2000 combinations of tasks, architectures, optimizers, and choices of LMs. Using LangProBe, we are the first to study the impact of program architectures and optimizers (and their compositions together and with different models) on tradeoffs of quality and cost. We find that optimized language programs offer strong cost--quality Pareto improvement over raw calls to models, but simultaneously demonstrate that human judgment (or empirical decisions) about which compositions to pursue is still necessary for best performance. We will open source the code and evaluation data for LangProBe. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_20315 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LangProBe: a Language Programs Benchmark Tan, Shangyin Agrawal, Lakshya A Singhvi, Arnav Lai, Liheng Ryan, Michael J Klein, Dan Khattab, Omar Sen, Koushik Zaharia, Matei Computation and Language Artificial Intelligence Information Retrieval Machine Learning Composing language models (LMs) into multi-step language programs and automatically optimizing their modular prompts is now a mainstream paradigm for building AI systems, but the tradeoffs in this space have only scarcely been studied before. We introduce LangProBe, the first large-scale benchmark for evaluating the architectures and optimization strategies for language programs, with over 2000 combinations of tasks, architectures, optimizers, and choices of LMs. Using LangProBe, we are the first to study the impact of program architectures and optimizers (and their compositions together and with different models) on tradeoffs of quality and cost. We find that optimized language programs offer strong cost--quality Pareto improvement over raw calls to models, but simultaneously demonstrate that human judgment (or empirical decisions) about which compositions to pursue is still necessary for best performance. We will open source the code and evaluation data for LangProBe. |
| title | LangProBe: a Language Programs Benchmark |
| topic | Computation and Language Artificial Intelligence Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2502.20315 |