Masked Language Modeling Becomes Conditional Density Estimation for Tabular Data Synthesis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911992367808512 |
|---|---|
| author | An, Seunghwan Woo, Gyeongdong Lim, Jaesung Kim, ChangHyun Hong, Sungchul Jeon, Jong-June |
| author_facet | An, Seunghwan Woo, Gyeongdong Lim, Jaesung Kim, ChangHyun Hong, Sungchul Jeon, Jong-June |
| contents | In this paper, our goal is to generate synthetic data for heterogeneous (mixed-type) tabular datasets with high machine learning utility (MLu). Since the MLu performance depends on accurately approximating the conditional distributions, we focus on devising a synthetic data generation method based on conditional distribution estimation. We introduce MaCoDE by redefining the consecutive multi-class classification task of Masked Language Modeling (MLM) as histogram-based non-parametric conditional density estimation. Our approach enables the estimation of conditional densities across arbitrary combinations of target and conditional variables. We bridge the theoretical gap between distributional learning and MLM by demonstrating that minimizing the orderless multi-class classification loss leads to minimizing the total variation distance between conditional distributions. To validate our proposed model, we evaluate its performance in synthetic data generation across 10 real-world datasets, demonstrating its ability to adjust data privacy levels easily without re-training. Additionally, since masked input tokens in MLM are analogous to missing data, we further assess its effectiveness in handling training datasets with missing values, including multiple imputations of the missing entries. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_20602 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Masked Language Modeling Becomes Conditional Density Estimation for Tabular Data Synthesis An, Seunghwan Woo, Gyeongdong Lim, Jaesung Kim, ChangHyun Hong, Sungchul Jeon, Jong-June Machine Learning Computation and Language In this paper, our goal is to generate synthetic data for heterogeneous (mixed-type) tabular datasets with high machine learning utility (MLu). Since the MLu performance depends on accurately approximating the conditional distributions, we focus on devising a synthetic data generation method based on conditional distribution estimation. We introduce MaCoDE by redefining the consecutive multi-class classification task of Masked Language Modeling (MLM) as histogram-based non-parametric conditional density estimation. Our approach enables the estimation of conditional densities across arbitrary combinations of target and conditional variables. We bridge the theoretical gap between distributional learning and MLM by demonstrating that minimizing the orderless multi-class classification loss leads to minimizing the total variation distance between conditional distributions. To validate our proposed model, we evaluate its performance in synthetic data generation across 10 real-world datasets, demonstrating its ability to adjust data privacy levels easily without re-training. Additionally, since masked input tokens in MLM are analogous to missing data, we further assess its effectiveness in handling training datasets with missing values, including multiple imputations of the missing entries. |
| title | Masked Language Modeling Becomes Conditional Density Estimation for Tabular Data Synthesis |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2405.20602 |