Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.10696 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912585454977024 |
|---|---|
| author | Wang, Shuaiqi Raunak, Vikas Backurs, Arturs Reis, Victor Zhou, Pei Chen, Sihao Yang, Longqi Lin, Zinan Yekhanin, Sergey Fanti, Giulia |
| author_facet | Wang, Shuaiqi Raunak, Vikas Backurs, Arturs Reis, Victor Zhou, Pei Chen, Sihao Yang, Longqi Lin, Zinan Yekhanin, Sergey Fanti, Giulia |
| contents | Differentially private (DP) synthetic data generation is a promising technique for utilizing private datasets that otherwise cannot be exposed for model training or other analytics. While much research literature has focused on generating private unstructured text and image data, in enterprise settings, structured data (e.g., tabular) is more common, often including natural language fields or components. Existing synthetic data evaluation techniques (e.g., FID) struggle to capture the structural properties and correlations of such datasets. In this work, we propose Struct-Bench, a framework and benchmark for evaluating synthetic datasets derived from structured datasets that contain natural language data. The Struct-Bench framework requires users to provide a representation of their dataset structure as a Context-Free Grammar (CFG). Our benchmark comprises 5 real-world and 2 synthetically generated datasets, each annotated with CFGs. We show that these datasets demonstrably present a great challenge even for state-of-the-art DP synthetic data generation methods. Struct-Bench also includes reference implementations of different metrics and a leaderboard, thereby providing researchers a standardized evaluation platform to benchmark and investigate privacy-preserving synthetic data generation methods. Further, we also present a case study showing how to use Struct-Bench to improve the synthetic data quality of Private Evolution (PE) on structured data. The benchmark and the leaderboard have been publicly made available at https://struct-bench.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_10696 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Struct-Bench: A Benchmark for Differentially Private Structured Text Generation Wang, Shuaiqi Raunak, Vikas Backurs, Arturs Reis, Victor Zhou, Pei Chen, Sihao Yang, Longqi Lin, Zinan Yekhanin, Sergey Fanti, Giulia Computation and Language Machine Learning Differentially private (DP) synthetic data generation is a promising technique for utilizing private datasets that otherwise cannot be exposed for model training or other analytics. While much research literature has focused on generating private unstructured text and image data, in enterprise settings, structured data (e.g., tabular) is more common, often including natural language fields or components. Existing synthetic data evaluation techniques (e.g., FID) struggle to capture the structural properties and correlations of such datasets. In this work, we propose Struct-Bench, a framework and benchmark for evaluating synthetic datasets derived from structured datasets that contain natural language data. The Struct-Bench framework requires users to provide a representation of their dataset structure as a Context-Free Grammar (CFG). Our benchmark comprises 5 real-world and 2 synthetically generated datasets, each annotated with CFGs. We show that these datasets demonstrably present a great challenge even for state-of-the-art DP synthetic data generation methods. Struct-Bench also includes reference implementations of different metrics and a leaderboard, thereby providing researchers a standardized evaluation platform to benchmark and investigate privacy-preserving synthetic data generation methods. Further, we also present a case study showing how to use Struct-Bench to improve the synthetic data quality of Private Evolution (PE) on structured data. The benchmark and the leaderboard have been publicly made available at https://struct-bench.github.io. |
| title | Struct-Bench: A Benchmark for Differentially Private Structured Text Generation |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2509.10696 |