DP-Bench: A Benchmark for Evaluating Data Product Creation Systems
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917153192542208 |
|---|---|
| author | Chowdhury, Faisal Shirai, Sola Dash, Sarthak Mihindukulasooriya, Nandana Samulowitz, Horst |
| author_facet | Chowdhury, Faisal Shirai, Sola Dash, Sarthak Mihindukulasooriya, Nandana Samulowitz, Horst |
| contents | A data product is created with the intention of solving a specific problem, addressing a specific business usecase or meeting a particular need, going beyond just serving data as a raw asset. Data products enable end users to gain greater insights about their data. Since it was first introduced over a decade ago, there has been considerable work, especially in industry, to create data products manually or semi-automatically. However, there exists hardly any benchmark to evaluate automatic data product creation. In this work, we present a benchmark, first of its kind, for this task. We call it DP-Bench. We describe how this benchmark was created by taking advantage of existing work in ELT (Extract-Load-Transform) and Text-to-SQL benchmarks. We also propose a number of LLM based approaches that can be considered as baselines for generating data products automatically. We make the DP-Bench and supplementary materials available in https://huggingface.co/datasets/ibm-research/dp-bench . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_15798 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DP-Bench: A Benchmark for Evaluating Data Product Creation Systems Chowdhury, Faisal Shirai, Sola Dash, Sarthak Mihindukulasooriya, Nandana Samulowitz, Horst Databases Computation and Language A data product is created with the intention of solving a specific problem, addressing a specific business usecase or meeting a particular need, going beyond just serving data as a raw asset. Data products enable end users to gain greater insights about their data. Since it was first introduced over a decade ago, there has been considerable work, especially in industry, to create data products manually or semi-automatically. However, there exists hardly any benchmark to evaluate automatic data product creation. In this work, we present a benchmark, first of its kind, for this task. We call it DP-Bench. We describe how this benchmark was created by taking advantage of existing work in ELT (Extract-Load-Transform) and Text-to-SQL benchmarks. We also propose a number of LLM based approaches that can be considered as baselines for generating data products automatically. We make the DP-Bench and supplementary materials available in https://huggingface.co/datasets/ibm-research/dp-bench . |
| title | DP-Bench: A Benchmark for Evaluating Data Product Creation Systems |
| topic | Databases Computation and Language |
| url | https://arxiv.org/abs/2512.15798 |